Alternatives and similar repositories for starcoder-truss

Users that are interested in starcoder-truss are comparing it to the libraries listed below

• basetenlabs / stablelm-truss
  ☆13Updated last year
• basetenlabs / truss
  The simplest way to serve AI/ML models in production
  ☆994Updated last week
• basetenlabs / truss-examples
  Examples of models deployable with Truss
  ☆174Updated this week
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆1,062Updated last year
• dstackai / dstack
  dstack is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML te…
  ☆1,797Updated this week
• ray-project / ray-llm
  RayLLM - LLMs on Ray (Archived). Read README for more info.
  ☆1,261Updated 2 months ago
• S-LoRA / S-LoRA
  S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  ☆1,830Updated last year
• aphrodite-engine / aphrodite-engine
  Large-scale LLM inference engine
  ☆1,440Updated last week
• efeslab / Nanoflow
  A throughput-oriented high-performance serving framework for LLMs
  ☆815Updated 3 weeks ago
• deepspeedai / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆2,019Updated 2 months ago
• predibase / lorax
  Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
  ☆2,996Updated 2 weeks ago
• huggingface / datatrove
  Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
  ☆2,396Updated last week
• pytorch / ao
  PyTorch native quantization and sparsity for training and inference
  ☆2,088Updated this week
• turboderp-org / exllamav2
  A fast inference library for running LLMs locally on modern consumer-class GPUs
  ☆4,202Updated this week
• huggingface / nanotron
  Minimalistic large language model 3D-parallelism training
  ☆1,909Updated last week
• tigr81 / tigr81
  A Project Scaffolder for HUMANS.
  ☆10Updated 7 months ago
• IST-DASLab / marlin
  FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
  ☆836Updated 9 months ago
• Cornell-RelaxML / quip-sharp
  ☆539Updated 7 months ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆332Updated this week
• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆796Updated 3 months ago
• featureform / featureform
  The Virtual Feature Store. Turn your existing data infrastructure into a feature store.
  ☆1,901Updated 3 weeks ago
• marella / ctransformers
  Python bindings for the Transformer models implemented in C/C++ using GGML library.
  ☆1,866Updated last year
• flashinfer-ai / flashinfer
  FlashInfer: Kernel Library for LLM Serving
  ☆3,123Updated this week
• flexflow / flexflow-train
  Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
  ☆1,799Updated this week
• intel / intel-extension-for-transformers
  ⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Pl…
  ☆2,170Updated 8 months ago
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆925Updated 6 months ago
• LMCache / LMCache
  Redis for LLMs
  ☆1,243Updated this week
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆845Updated this week
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆444Updated last week
• vllm-project / llm-compressor
  Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
  ☆1,441Updated this week