Alternatives and similar repositories for trt-llm-as-openai-windows:

Users that are interested in trt-llm-as-openai-windows are comparing it to the libraries listed below

• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆191Updated this week
• NVIDIA / RTX-AI-Toolkit
  The NVIDIA RTX™ AI Toolkit is a suite of tools and SDKs for Windows developers to customize, optimize, and deploy AI models across RTX PC…
  ☆141Updated 3 months ago
• intel / neural-speed
  An innovative library for efficient LLM inference via low-bit quantization
  ☆351Updated 5 months ago
• tdrussell / qlora-pipe
  A pipeline parallel training script for LLMs.
  ☆124Updated 3 weeks ago
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆88Updated this week
• nyunAI / PruneGPT
  ☆53Updated 8 months ago
• cognitivecomputations / kraken
  ☆65Updated 8 months ago
• LLM360 / amber-data-prep
  Data preparation code for Amber 7B LLM
  ☆85Updated 9 months ago
• willccbb / mlx_parallm
  Fast parallel LLM inference for MLX
  ☆163Updated 7 months ago
• astramind-ai / BitMat
  An efficent implementation of the method proposed in "The Era of 1-bit LLMs"
  ☆154Updated 4 months ago
• cognitivecomputations / spectrum
  ☆113Updated 4 months ago
• taprosoft / llm_finetuning
  Convenient wrapper for fine-tuning and inference of Large Language Models (LLMs) with several quantization techniques (GTPQ, bitsandbytes…
  ☆147Updated last year
• cognitivecomputations / laserRMT
  This is our own implementation of 'Layer Selective Rank Reduction'
  ☆233Updated 8 months ago
• VITA-Group / Q-GaLore
  Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
  ☆191Updated 7 months ago
• leafspark / AutoGGUF
  automatically quant GGUF models
  ☆154Updated this week
• thomasgauthier / LoRD
  Low-Rank adapter extraction for fine-tuned transformers models
  ☆169Updated 9 months ago
• arcee-ai / EvolKit
  EvolKit is an innovative framework designed to automatically enhance the complexity of instructions used for fine-tuning Large Language M…
  ☆203Updated 3 months ago
• BrunoGeorgevich / llama3.cp
  Adapted version of llama3.np (NumPy) to a CuPy implementation for the Llama 3 model.
  ☆36Updated 9 months ago
• cognitivecomputations / grokadamw
  ☆123Updated 6 months ago
• sdan / selfextend
  an implementation of Self-Extend, to expand the context window via grouped attention
  ☆118Updated last year
• writer / writing-in-the-margins
  ☆109Updated 5 months ago
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆193Updated 6 months ago
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆136Updated 6 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 8 months ago
• Aesthisia / LLMinator
  Gradio based tool to run opensource LLM models directly from Huggingface
  ☆91Updated 7 months ago
• AlexBodner / How_Much_VRAM
  ☆98Updated 5 months ago
• huggingface / optimum-benchmark
  🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…
  ☆288Updated 3 weeks ago
• run-ai / runai-model-streamer
  ☆161Updated this week
• epolewski / EricLLM
  A fast batching API to serve LLM models
  ☆180Updated 9 months ago
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆193Updated 2 weeks ago