Alternatives and similar repositories for runai-model-streamer

Users that are interested in runai-model-streamer are comparing it to the libraries listed below

• coreweave / tensorizer
  Module, Model, and Tensor Serialization/Deserialization
  ☆232Updated last week
• NVIDIA / cuda-checkpoint
  CUDA checkpoint and restore utility
  ☆339Updated 4 months ago
• huggingface / inference-benchmarker
  Inference server benchmarking tool
  ☆64Updated last month
• triton-inference-server / vllm_backend
  ☆260Updated 2 weeks ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆308Updated last week
• IBM / text-generation-inference
  IBM development fork of https://github.com/huggingface/text-generation-inference
  ☆60Updated 3 weeks ago
• coreweave / ml-containers
  ☆34Updated last week
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆127Updated last month
• NVIDIA / nvidia-resiliency-ext
  NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the …
  ☆169Updated this week
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆207Updated 10 months ago
• imbue-ai / cluster-health
  ☆308Updated 9 months ago
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆262Updated 7 months ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆75Updated 2 weeks ago
• foundation-model-stack / fastsafetensors
  High-performance safetensors model loader
  ☆34Updated last week
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆137Updated 10 months ago
• pytorch / torchft
  PyTorch per step fault tolerance (actively under development)
  ☆302Updated this week
• triton-inference-server / triton_distributed
  ☆49Updated 2 months ago
• triton-inference-server / triton_cli
  Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inferen…
  ☆62Updated 2 weeks ago
• runpod-workers / worker-sglang
  SGLang is fast serving framework for large language models and vision language models.
  ☆23Updated 3 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆112Updated 4 months ago
• ppl-ai / pplx-kernels
  Perplexity GPU Kernels
  ☆318Updated last week
• kserve / open-inference-protocol
  Repository for open inference protocol specification
  ☆56Updated 2 weeks ago
• huggingface / optimum-benchmark
  🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…
  ☆301Updated this week
• opendatahub-io / vllm-tgis-adapter
  vLLM adapter for a TGIS-compatible gRPC server.
  ☆30Updated this week
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆67Updated last year
• huggingface / gpu-fryer
  Where GPUs get cooked 👩‍🍳🔥
  ☆229Updated 2 months ago
• leptonai / gpud
  GPUd automates monitoring, diagnostics, and issue identification for GPUs
  ☆362Updated this week
• NVIDIA / KAI-Scheduler
  KAI Scheduler is an open source Kubernetes Native scheduler for AI workloads at large scale
  ☆590Updated this week
• mani-kantap / llm-inference-solutions
  A collection of all available inference solutions for the LLMs
  ☆88Updated 3 months ago
• huggingface / xet-core
  xet client tech, used in huggingface_hub
  ☆103Updated last week