Alternatives and similar repositories for mosec:

Users that are interested in mosec are comparing it to the libraries listed below

• triton-inference-server / fastertransformer_backend
  ☆411Updated last year
• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆786Updated 2 months ago
• deepspeedai / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆2,002Updated 3 weeks ago
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆1,054Updated 11 months ago
• triton-inference-server / python_backend
  Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
  ☆600Updated this week
• ELS-RD / transformer-deploy
  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
  ☆1,684Updated 5 months ago
• ray-project / ray-llm
  RayLLM - LLMs on Ray (Archived). Read README for more info.
  ☆1,263Updated last month
• microsoft / Llama-2-Onnx
  ☆1,025Updated last year
• triton-inference-server / tutorials
  This repository contains tutorials and examples for Triton Inference Server
  ☆682Updated 3 weeks ago
• triton-inference-server / model_analyzer
  Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…
  ☆470Updated last month
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆868Updated 4 months ago
• ELS-RD / kernl
  Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackab…
  ☆1,564Updated last year
• tensorchord / openmodelz
  Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
  ☆263Updated last year
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆561Updated 6 months ago
• flexflow / flexflow-train
  Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
  ☆1,783Updated last week
• BaguaSys / bagua
  Bagua Speeds up PyTorch
  ☆879Updated 8 months ago
• qdrant / quaterion
  Blazing fast framework for fine-tuning similarity learning models
  ☆657Updated last week
• pytorch / torchx
  TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and sup…
  ☆358Updated this week
• hpcaitech / EnergonAI
  Large-scale model inference.
  ☆628Updated last year
• salesforce / xgen
  Salesforce open-source LLMs with 8k sequence length.
  ☆717Updated 2 months ago
• tensorchord / envd
  🏕️ Reproducible development environment
  ☆2,110Updated this week
• huggingface / optimum
  🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization…
  ☆2,845Updated this week
• triton-inference-server / backend
  Common source, scripts and utilities for creating Triton backends.
  ☆315Updated this week
• mlc-ai / xgrammar
  Fast, Flexible and Portable Structured Generation
  ☆878Updated last week
• SeldonIO / MLServer
  An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
  ☆794Updated this week
• S-LoRA / S-LoRA
  S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  ☆1,816Updated last year
• ray-project / llmperf-leaderboard
  ☆455Updated last year
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆430Updated this week
• triton-inference-server / vllm_backend
  ☆241Updated this week
• skeskinen / bert.cpp
  ggml implementation of BERT
  ☆485Updated last year