Alternatives and similar repositories for runai-model-streamer:

Users that are interested in runai-model-streamer are comparing it to the libraries listed below

• coreweave / tensorizer
  Module, Model, and Tensor Serialization/Deserialization
  ☆221Updated last month
• NVIDIA / cuda-checkpoint
  CUDA checkpoint and restore utility
  ☆322Updated 2 months ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆248Updated this week
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆70Updated 2 months ago
• leptonai / gpud
  GPUd automates monitoring, diagnostics, and issue identification for GPUs
  ☆330Updated this week
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆262Updated 6 months ago
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆124Updated last week
• huggingface / inference-benchmarker
  Inference server benchmarking tool
  ☆48Updated last week
• ppl-ai / pplx-kernels
  Perplexity GPU Kernels
  ☆185Updated this week
• pytorch / torchft
  PyTorch per step fault tolerance (actively under development)
  ☆273Updated this week
• IBM / text-generation-inference
  IBM development fork of https://github.com/huggingface/text-generation-inference
  ☆60Updated 3 months ago
• NVIDIA / nvidia-resiliency-ext
  NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the …
  ☆136Updated last week
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆204Updated 8 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆107Updated 2 months ago
• apple / ml-recurrent-drafter
  ☆205Updated 2 months ago
• bentoml / BentoVLLM
  Self-host LLMs with vLLM and BentoML
  ☆100Updated this week
• NVIDIA / KAI-Scheduler
  KAI Scheduler is an open source Kubernetes Native scheduler for AI workloads at large scale
  ☆450Updated this week
• AI-Hypercomputer / JetStream
  JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs wel…
  ☆313Updated this week
• imbue-ai / cluster-health
  ☆301Updated 7 months ago
• coreweave / ml-containers
  ☆30Updated 2 weeks ago
• triton-inference-server / vllm_backend
  ☆241Updated this week
• opendatahub-io / vllm-tgis-adapter
  vLLM adapter for a TGIS-compatible gRPC server.
  ☆25Updated this week
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆135Updated 8 months ago
• NVIDIA / NeMo-Run
  A tool to configure, launch and manage your machine learning experiments.
  ☆135Updated this week
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆238Updated this week
• huggingface / gpu-fryer
  Where GPUs get cooked 👩‍🍳🔥
  ☆221Updated last month
• NVIDIA / kvpress
  LLM KV cache compression made easy
  ☆452Updated 3 weeks ago
• run-ai / rntop
  A top-like tool for monitoring GPUs in a cluster
  ☆86Updated last year
• neuralmagic / compressed-tensors
  A safetensors extension to efficiently store sparse quantized tensors on disk
  ☆98Updated this week