neuralmagic / guidellm
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
☆167Updated 2 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for guidellm
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆134Updated 3 months ago
- experiments with inference on llama☆105Updated 5 months ago
- An Open Source Toolkit For LLM Distillation☆359Updated 2 months ago
- EvolKit is an innovative framework designed to automatically enhance the complexity of instructions used for fine-tuning Large Language M…☆181Updated 3 weeks ago
- Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM☆691Updated this week
- ☆123Updated this week
- Self-host LLMs with vLLM and BentoML☆74Updated last week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆253Updated last month
- ☆94Updated 2 months ago
- awesome synthetic (text) datasets☆243Updated 3 weeks ago
- Production ready LLM model compression/quantization toolkit with accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.☆126Updated this week
- Manage scalable open LLM inference endpoints in Slurm clusters☆238Updated 4 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆173Updated 4 months ago
- A collection of all available inference solutions for the LLMs☆73Updated 2 months ago
- ☆200Updated 9 months ago
- Tutorial for building LLM router☆163Updated 4 months ago
- Advanced Quantization Algorithm for LLMs. This is official implementation of "Optimize Weight Rounding via Signed Gradient Descent for t…☆248Updated this week
- Benchmark various LLM Structured Output frameworks: Instructor, Mirascope, Langchain, LlamaIndex, Fructose, Marvin, Outlines, etc on task…☆134Updated 2 months ago
- ☆130Updated this week
- This project showcases an LLMOps pipeline that fine-tunes a small-size LLM model to prepare for the outage of the service LLM.☆290Updated 2 months ago
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆196Updated 7 months ago
- Synthetic Data for LLM Fine-Tuning☆97Updated 11 months ago
- Let's build better datasets, together!☆206Updated this week
- Generate Synthetic Data Using OpenAI, MistralAI or AnthropicAI☆221Updated 6 months ago
- ☆193Updated this week
- ☆112Updated this week
- Late Interaction Models Training & Retrieval☆166Updated this week
- A pipeline for LLM knowledge distillation☆78Updated 3 months ago
- ☆133Updated 4 months ago
- Banishing LLM Hallucinations Requires Rethinking Generalization☆261Updated 4 months ago