Alternatives and similar repositories for trt-llm-as-openai-windows:

Users that are interested in trt-llm-as-openai-windows are comparing it to the libraries listed below

• epolewski / EricLLM
  A fast batching API to serve LLM models
  ☆177Updated 8 months ago
• tdrussell / qlora-pipe
  A pipeline parallel training script for LLMs.
  ☆116Updated this week
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆183Updated last month
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆138Updated 5 months ago
• cognitivecomputations / kraken
  ☆65Updated 7 months ago
• cognitivecomputations / OpenChatML
  ☆151Updated 6 months ago
• NVIDIA / RTX-AI-Toolkit
  The NVIDIA RTX™ AI Toolkit is a suite of tools and SDKs for Windows developers to customize, optimize, and deploy AI models across RTX PC…
  ☆126Updated last month
• Cerebras / DocChat
  GPT-4 Level Conversational QA Trained In a Few Hours
  ☆58Updated 4 months ago
• thomasgauthier / LoRD
  Low-Rank adapter extraction for fine-tuned transformers models
  ☆165Updated 8 months ago
• run-ai / runai-model-streamer
  ☆150Updated this week
• cognitivecomputations / spectrum
  ☆108Updated 3 months ago
• TheBlokeAI / AIScripts
  Some simple scripts that I use day-to-day when working with LLMs and Huggingface Hub
  ☆156Updated last year
• anyscale / llm-router
  Tutorial for building LLM router
  ☆170Updated 5 months ago
• kerekovskik / autologic
  autologic is a Python package that implements the SELF-DISCOVER framework proposed in the paper SELF-DISCOVER: Large Language Models Self…
  ☆57Updated 10 months ago
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆88Updated this week
• cognitivecomputations / grokadamw
  ☆122Updated 4 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 7 months ago
• cognitivecomputations / laserRMT
  This is our own implementation of 'Layer Selective Rank Reduction'
  ☆232Updated 7 months ago
• janhq / cortex.tensorrt-llm
  Cortex.Tensorrt-LLM is a C++ inference library that can be loaded by any server at runtime. It submodules NVIDIA’s TensorRT-LLM for GPU a…
  ☆42Updated 3 months ago
• lamini-ai / Lamini-Memory-Tuning
  Banishing LLM Hallucinations Requires Rethinking Generalization
  ☆268Updated 6 months ago
• teknium1 / ShareGPT-Builder
  ☆107Updated 3 weeks ago
• michaelfeil / embed
  A stable, fast and easy-to-use inference library with a focus on a sync-to-async API
  ☆44Updated 3 months ago
• Vaibhavs10 / optimise-my-whisper
  ☆195Updated 7 months ago
• leafspark / AutoGGUF
  automatically quant GGUF models
  ☆150Updated this week
• cfahlgren1 / observers
  A Lightweight Library for AI Observability
  ☆229Updated this week
• h2oai / enterprise-h2ogpte
  Client Code Examples, Use Cases and Benchmarks for Enterprise h2oGPTe RAG-Based GenAI Platform
  ☆82Updated last week
• astramind-ai / BitMat
  An efficent implementation of the method proposed in "The Era of 1-bit LLMs"
  ☆154Updated 3 months ago
• NVIDIA / workbench-example-hybrid-rag
  An NVIDIA AI Workbench example project for Retrieval Augmented Generation (RAG)
  ☆289Updated last month
• unslothai / unsloth-zoo
  Utils for Unsloth
  ☆27Updated this week
• sdan / selfextend
  an implementation of Self-Extend, to expand the context window via grouped attention
  ☆118Updated last year