Alternatives and similar repositories for mosec

Users that are interested in mosec are comparing it to the libraries listed below

• deepspeedai / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆2,020Updated 2 months ago
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆1,066Updated last year
• S-LoRA / S-LoRA
  S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  ☆1,835Updated last year
• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆801Updated 4 months ago
• triton-inference-server / fastertransformer_backend
  ☆411Updated last year
• intel / intel-extension-for-transformers
  ⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Pl…
  ☆2,169Updated 8 months ago
• huggingface / optimum
  🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization…
  ☆2,942Updated this week
• tensorchord / openmodelz
  Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
  ☆269Updated last year
• srush / llama2.rs
  A fast llama2 decoder in pure Rust.
  ☆1,051Updated last year
• hao-ai-lab / LookaheadDecoding
  [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  ☆1,258Updated 3 months ago
• ray-project / ray-llm
  RayLLM - LLMs on Ray (Archived). Read README for more info.
  ☆1,261Updated 3 months ago
• ELS-RD / transformer-deploy
  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
  ☆1,688Updated 8 months ago
• flexflow / flexflow-train
  Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
  ☆1,803Updated this week
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆448Updated this week
• predibase / lorax
  Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
  ☆3,024Updated last month
• BlackSamorez / tensor_parallel
  Automatically split your PyTorch models on multiple GPUs for training & inference
  ☆655Updated last year
• huggingface / optimum-benchmark
  🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…
  ☆304Updated 3 weeks ago
• ELS-RD / kernl
  Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackab…
  ☆1,571Updated last year
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆940Updated 6 months ago
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆851Updated this week
• huggingface / nanotron
  Minimalistic large language model 3D-parallelism training
  ☆1,926Updated last week
• skeskinen / bert.cpp
  ggml implementation of BERT
  ☆493Updated last year
• triton-inference-server / python_backend
  Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
  ☆619Updated last week
• efeslab / Nanoflow
  A throughput-oriented high-performance serving framework for LLMs
  ☆825Updated 2 weeks ago
• Vahe1994 / SpQR
  ☆543Updated 6 months ago
• casper-hansen / AutoAWQ
  AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:
  ☆2,193Updated last month
• crabml / crabml
  a fast cross platform AI inference engine 🤖 using Rust 🦀 and WebGPU 🎮
  ☆451Updated 5 months ago
• FasterDecoding / Medusa
  Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
  ☆2,549Updated 11 months ago
• hpcaitech / SwiftInfer
  Efficient AI Inference & Serving
  ☆471Updated last year
• pytorch / torchdynamo
  A Python-level JIT compiler designed to make unmodified PyTorch programs faster.
  ☆1,051Updated last year