Alternatives and similar repositories for mosec:

Users that are interested in mosec are comparing it to the libraries listed below

• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆790Updated 2 months ago
• ray-project / ray-llm
  RayLLM - LLMs on Ray (Archived). Read README for more info.
  ☆1,261Updated last month
• triton-inference-server / fastertransformer_backend
  ☆411Updated last year
• deepspeedai / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆2,009Updated last month
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆832Updated this week
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆561Updated 7 months ago
• hpcaitech / EnergonAI
  Large-scale model inference.
  ☆629Updated last year
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆1,058Updated last year
• triton-inference-server / python_backend
  Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
  ☆608Updated this week
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆437Updated this week
• ELS-RD / transformer-deploy
  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
  ☆1,683Updated 6 months ago
• casper-hansen / AutoAWQ
  AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:
  ☆2,145Updated last week
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,386Updated last year
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆900Updated 5 months ago
• triton-inference-server / tutorials
  This repository contains tutorials and examples for Triton Inference Server
  ☆695Updated 3 weeks ago
• huggingface / hf_transfer
  ☆462Updated last month
• hpcaitech / SwiftInfer
  Efficient AI Inference & Serving
  ☆470Updated last year
• BaguaSys / bagua
  Bagua Speeds up PyTorch
  ☆883Updated 9 months ago
• michaelfeil / infinity
  Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
  ☆2,138Updated 3 weeks ago
• microsoft / Llama-2-Onnx
  ☆1,026Updated last year
• bentoml / Yatai
  Model Deployment at Scale on Kubernetes 🦄️
  ☆811Updated last year
• triton-inference-server / model_analyzer
  Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…
  ☆474Updated 2 weeks ago
• kuleshov-group / llmtools
  Finetuning Large Language Models on One Consumer GPU in 2 Bits
  ☆723Updated 11 months ago
• RWKV / rwkv.cpp
  INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
  ☆1,514Updated last month
• jina-ai / vectordb
  A Python vector database you just need - no more, no less.
  ☆610Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆205Updated 9 months ago
• mit-han-lab / llm-awq
  [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  ☆2,991Updated this week
• flexflow / flexflow-train
  Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
  ☆1,792Updated this week
• AetherCortex / Llama-X
  Open Academic Research on Improving LLaMA to SOTA LLM
  ☆1,620Updated last year
• mbzuai-nlp / LaMini-LM
  LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
  ☆820Updated 2 years ago