Alternatives and similar repositories for text-generation-inference:

Users that are interested in text-generation-inference are comparing it to the libraries listed below

• opendatahub-io / vllm-tgis-adapter
  vLLM adapter for a TGIS-compatible gRPC server.
  ☆27Updated this week
• foundation-model-stack / fms-hf-tuning
  🚀 Collection of tuning recipes with HuggingFace SFTTrainer and PyTorch FSDP.
  ☆41Updated this week
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆70Updated 2 months ago
• IBM / dolomite-engine
  Dolomite Engine is a library for pretraining/finetuning LLMs
  ☆52Updated this week
• run-ai / runai-model-streamer
  ☆207Updated this week
• instructlab / sdg
  Python library for Synthetic Data Generation
  ☆42Updated this week
• huggingface / inference-benchmarker
  Inference server benchmarking tool
  ☆56Updated last week
• instructlab / training
  InstructLab Training Library - Efficient Fine-Tuning with Message-Format Data
  ☆38Updated this week
• substratusai / vllm-docker
  ☆59Updated last month
• huggingface / tgi-gaudi
  Large Language Model Text Generation Inference on Habana Gaudi
  ☆33Updated last month
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 11 months ago
• bentoml / llm-bench
  ☆50Updated 5 months ago
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆263Updated 6 months ago
• snowflakedb / ArcticTraining
  ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)
  ☆86Updated this week
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆115Updated last week
• nyunAI / PruneGPT
  ☆53Updated 11 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆109Updated 3 months ago
• snowflakedb / ArcticInference
  ☆45Updated last week
• runpod-workers / worker-sglang
  SGLang is fast serving framework for large language models and vision language models.
  ☆22Updated 2 months ago
• SqueezeAILab / Tool2Vec
  Efficient and Scalable Estimation of Tool Representations in Vector Space
  ☆23Updated 8 months ago
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆136Updated 9 months ago
• samchaineau / llm_slerp_generation
  Repo hosting codes and materials related to speeding LLMs' inference using token merging.
  ☆36Updated last year
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆83Updated last month
• IBM / unitxt
  🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data …
  ☆191Updated this week
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆116Updated 5 months ago
• cognitivecomputations / kraken
  ☆66Updated 11 months ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆284Updated this week
• abacusai / gh200-llm
  Docker image NVIDIA GH200 machines - optimized for vllm serving and hf trainer finetuning
  ☆40Updated 2 months ago
• Snowflake-Labs / vllm
  ☆15Updated last month
• facebookresearch / matrix
  Matrix (Multi-Agent daTa geneRation Infra and eXperimentation framework) is a versatile engine for multi-agent conversational data genera…
  ☆29Updated this week