Alternatives and similar repositories for runai-model-streamer:

Users that are interested in runai-model-streamer are comparing it to the libraries listed below

• coreweave / tensorizer
  Module, Model, and Tensor Serialization/Deserialization
  ☆217Updated last month
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆224Updated this week
• pytorch / torchft
  PyTorch per step fault tolerance (actively under development)
  ☆266Updated this week
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆262Updated 5 months ago
• bentoml / BentoVLLM
  Self-host LLMs with vLLM and BentoML
  ☆94Updated this week
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆69Updated last month
• apple / ml-recurrent-drafter
  ☆203Updated last month
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆122Updated 3 weeks ago
• NVIDIA / cuda-checkpoint
  CUDA checkpoint and restore utility
  ☆306Updated last month
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆136Updated 7 months ago
• IBM / text-generation-inference
  IBM development fork of https://github.com/huggingface/text-generation-inference
  ☆60Updated 3 months ago
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated 11 months ago
• NVIDIA / NeMo-Run
  A tool to configure, launch and manage your machine learning experiments.
  ☆127Updated this week
• triton-inference-server / vllm_backend
  ☆237Updated last week
• AI-Hypercomputer / JetStream
  JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs wel…
  ☆297Updated this week
• astramind-ai / BitMat
  An efficent implementation of the method proposed in "The Era of 1-bit LLMs"
  ☆154Updated 5 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆103Updated 2 months ago
• imbue-ai / cluster-health
  ☆296Updated 7 months ago
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆87Updated this week
• run-ai / llmperf
  ☆54Updated 6 months ago
• NVIDIA / kvpress
  LLM KV cache compression made easy
  ☆440Updated this week
• run-ai / rntop
  A top-like tool for monitoring GPUs in a cluster
  ☆86Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆201Updated 7 months ago
• Preemo-Inc / text-generation-inference
  ☆199Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆109Updated 3 months ago
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆232Updated 2 weeks ago
• mani-kantap / llm-inference-solutions
  A collection of all available inference solutions for the LLMs
  ☆81Updated 3 weeks ago
• ServerlessLLM / ServerlessLLM
  Serverless LLM Serving for Everyone.
  ☆437Updated this week
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆86Updated 2 months ago
• leptonai / gpud
  GPUd automates monitoring, diagnostics, and issue identification for GPUs
  ☆286Updated this week