Alternatives and similar repositories for olah

Users that are interested in olah are comparing it to the libraries listed below

• run-ai / runai-model-streamer
  ☆207Updated last week
• gpustack / llama-box
  LM inference server implementation based on *.cpp.
  ☆185Updated last week
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated last year
• tensorchord / deepseek-api-arena
  A benchmarking tool for comparing different LLM API providers' DeepSeek model deployments.
  ☆29Updated last month
• gpustack / vox-box
  A text-to-speech and speech-to-text server compatible with the OpenAI API, supporting Whisper, FunASR, Bark, and CosyVoice backends.
  ☆110Updated this week
• bentoml / BentoVLLM
  Self-host LLMs with vLLM and BentoML
  ☆109Updated last week
• clearml / clearml-fractional-gpu
  ClearML Fractional GPU - Run multiple containers on the same GPU with driver level memory limitation ✨ and compute time-slicing
  ☆78Updated 9 months ago
• tensorchord / openmodelz
  Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
  ☆265Updated last year
• leptonai / gpud
  GPUd automates monitoring, diagnostics, and issue identification for GPUs
  ☆352Updated this week
• gpustack / gguf-parser-go
  Review/Check GGUF files and estimate the memory usage and maximum tokens per second.
  ☆161Updated this week
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆205Updated 9 months ago
• huggingface / xet-core
  xet client tech, used in huggingface_hub
  ☆95Updated this week
• NVIDIA / gpu-driver-container
  The NVIDIA GPU driver container allows the provisioning of the NVIDIA driver through the use of containers.
  ☆111Updated last week
• Yard1 / Ray-DeepSpeed-Inference
  ☆17Updated 2 years ago
• limcheekin / open-text-embeddings
  Open Source Text Embedding Models with OpenAI Compatible API
  ☆153Updated 10 months ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆291Updated this week
• bentoml / llm-bench
  ☆50Updated 5 months ago
• nvwacloud / tensorlink
  Unlock Unlimited Potential! Share Your GPU Power Across Your Local Network!
  ☆56Updated 10 months ago
• coreweave / tensorizer
  Module, Model, and Tensor Serialization/Deserialization
  ☆227Updated this week
• substratusai / vllm-docker
  ☆59Updated last month
• NVIDIA / vgpu-device-manager
  NVIDIA vGPU Device Manager manages NVIDIA vGPU devices on top of Kubernetes
  ☆130Updated last week
• chaunceyjiang / fake-gpu
  This project is designed to simulate GPU information, making it easier to test scenarios where a GPU is not available.
  ☆44Updated 2 months ago
• run-ai / fake-gpu-operator
  ☆110Updated last week
• coreweave / ml-containers
  ☆31Updated this week
• nekomeowww / ollama-operator
  🚢 Yet another operator for running large language models on Kubernetes with ease. Powered by Ollama! 🐫
  ☆187Updated this week
• matpool / mpu
  A shim driver allows in-docker nvidia-smi showing correct process list without modify anything
  ☆84Updated 2 months ago
• substratusai / stapi
  Sentence Transformers API: An OpenAI compatible embedding API server
  ☆58Updated 8 months ago
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆217Updated 4 months ago
• MooreThreads / csi-driver-3fs
  ☆11Updated 2 months ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆72Updated this week