Alternatives and similar repositories for cortex.tensorrt-llm

Users that are interested in cortex.tensorrt-llm are comparing it to the libraries listed below

• chigkim / Ollama-MMLU-Pro
  ☆107Updated 3 months ago
• janhq / cortex.llamacpp
  cortex.llamacpp is a high-efficiency C++ inference engine for edge computing. It is a dynamic library that can be loaded by any server a…
  ☆41Updated 4 months ago
• leafspark / AutoGGUF
  automatically quant GGUF models
  ☆214Updated last month
• NVIDIA / trt-llm-as-openai-windows
  This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…
  ☆127Updated last year
• Codys12 / airllm
  AirLLM 70B inference with single 4GB GPU
  ☆14Updated 5 months ago
• monk1337 / auto-ollama
  run ollama & gguf easily with a single command
  ☆52Updated last year
• the-crypt-keeper / tcurtsni
  Tcurtsni: Reverse Instruction Chat, ever wonder what your LLM wants to ask you?
  ☆23Updated last year
• bjj / exllamav2-openai-server
  An OpenAI API compatible LLM inference server based on ExLlamaV2.
  ☆25Updated last year
• unslothai / llama.cpp
  LLM inference in C/C++
  ☆103Updated 3 weeks ago
• huseinzol05 / transformers-continuous-batching
  Lightweight continuous batching OpenAI compatibility using HuggingFace Transformers include T5 and Whisper.
  ☆29Updated 8 months ago
• jllllll / exllama
  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
  ☆64Updated 2 years ago
• kevkid / gguf_gui
  ☆125Updated last year
• epolewski / EricLLM
  A fast batching API to serve LLM models
  ☆189Updated last year
• rafacelente / bllama
  1.58-bit LLaMa model
  ☆83Updated last year
• RandomInternetPreson / Lucid_Autonomy
  An extension that lets the AI take the wheel, allowing it to use the mouse and keyboard, recognize UI elements, and prompt itself :3...no…
  ☆128Updated last year
• Aesthisia / LLMinator
  Gradio based tool to run opensource LLM models directly from Huggingface
  ☆96Updated last year
• rodrigobaron / anthill
  ☆24Updated 10 months ago
• elgatopanzon / gatogpt
  Local LLM inference & management server with built-in OpenAI API
  ☆31Updated last year
• jukofyork / transplant-vocab
  Transplants vocabulary between language models, enabling the creation of draft models for speculative decoding WITHOUT retraining.
  ☆46Updated last month
• nyunAI / PruneGPT
  ☆51Updated last year
• fairydreaming / llama.cpp
  LLM inference in C/C++
  ☆21Updated 8 months ago
• fairydreaming / farel-bench
  Testing LLM reasoning abilities with family relationship quizzes.
  ☆63Updated 10 months ago
• Fus3n / TwoAI
  A simple experiment on letting two local LLM have a conversation about anything!
  ☆112Updated last year
• NVIDIA / RTX-AI-Toolkit
  The NVIDIA RTX™ AI Toolkit is a suite of tools and SDKs for Windows developers to customize, optimize, and deploy AI models across RTX PC…
  ☆179Updated this week
• xhedit / quantkit
  cli tool to quantize gguf, gptq, awq, hqq and exl2 models
  ☆76Updated 11 months ago
• abgulati / hf-waitress
  Serving LLMs in the HF-Transformers format via a PyFlask API
  ☆71Updated last year
• rombodawg / Easy_training
  ☆51Updated 9 months ago
• sanctuary-systems-com / llama_multiserver
  A proxy that hosts multiple single-model runners such as LLama.cpp and vLLM
  ☆12Updated 6 months ago
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆235Updated 11 months ago
• beyondExp / B-Llama3-o
  B-Llama3o a llama3 with Vision Audio and Audio understanding as well as text and Audio and Animation Data output.
  ☆26Updated last year