Alternatives and similar repositories for mosec

Users that are interested in mosec are comparing it to the libraries listed below

• triton-inference-server / fastertransformer_backend
  ☆411Updated last year
• deepspeedai / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆2,009Updated 2 months ago
• ray-project / ray-llm
  RayLLM - LLMs on Ray (Archived). Read README for more info.
  ☆1,260Updated 2 months ago
• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆791Updated 3 months ago
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆1,060Updated last year
• triton-inference-server / python_backend
  Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
  ☆615Updated 2 weeks ago
• ELS-RD / transformer-deploy
  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
  ☆1,685Updated 7 months ago
• hpcaitech / EnergonAI
  Large-scale model inference.
  ☆630Updated last year
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆842Updated this week
• ELS-RD / kernl
  Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackab…
  ☆1,568Updated last year
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆922Updated 5 months ago
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆564Updated 7 months ago
• hpcaitech / SwiftInfer
  Efficient AI Inference & Serving
  ☆469Updated last year
• triton-inference-server / model_analyzer
  Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…
  ☆476Updated last month
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆207Updated 10 months ago
• tensorchord / openmodelz
  Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
  ☆267Updated last year
• triton-inference-server / client
  Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
  ☆623Updated 2 weeks ago
• BlackSamorez / tensor_parallel
  Automatically split your PyTorch models on multiple GPUs for training & inference
  ☆653Updated last year
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,391Updated last year
• flexflow / flexflow-train
  Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
  ☆1,797Updated this week
• efeslab / Nanoflow
  A throughput-oriented high-performance serving framework for LLMs
  ☆814Updated 3 weeks ago
• huggingface / text-embeddings-inference
  A blazing fast inference solution for text embeddings models
  ☆3,624Updated this week
• michaelfeil / infinity
  Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
  ☆2,202Updated 2 weeks ago
• vllm-project / llm-compressor
  Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
  ☆1,417Updated this week
• marella / ctransformers
  Python bindings for the Transformer models implemented in C/C++ using GGML library.
  ☆1,864Updated last year
• skeskinen / bert.cpp
  ggml implementation of BERT
  ☆491Updated last year
• microsoft / Llama-2-Onnx
  ☆1,025Updated last year
• mlc-ai / xgrammar
  Fast, Flexible and Portable Structured Generation
  ☆980Updated last week
• BaguaSys / bagua
  Bagua Speeds up PyTorch
  ☆883Updated 10 months ago
• arielnlee / Platypus
  Code for fine-tuning Platypus fam LLMs using LoRA
  ☆628Updated last year