Alternatives and similar repositories for olah

Users that are interested in olah are comparing it to the libraries listed below

• run-ai / runai-model-streamer
  ☆222Updated this week
• gpustack / llama-box
  LM inference server implementation based on *.cpp.
  ☆226Updated this week
• matpool / mpu
  A shim driver allows in-docker nvidia-smi showing correct process list without modify anything
  ☆87Updated 3 months ago
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆209Updated 10 months ago
• huggingface / hf_transfer
  ☆472Updated 2 months ago
• ubergarm / r1-ktransformers-guide
  run DeepSeek-R1 GGUFs on KTransformers
  ☆236Updated 3 months ago
• huggingface / xet-core
  xet client tech, used in huggingface_hub
  ☆118Updated last week
• tensorchord / openmodelz
  Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
  ☆269Updated last year
• tensorchord / deepseek-api-arena
  A benchmarking tool for comparing different LLM API providers' DeepSeek model deployments.
  ☆28Updated 2 months ago
• MooreThreads / csi-driver-3fs
  ☆13Updated 3 months ago
• gpustack / gguf-parser-go
  Review/Check GGUF files and estimate the memory usage and maximum tokens per second.
  ☆177Updated last week
• NVIDIA / vgpu-device-manager
  NVIDIA vGPU Device Manager manages NVIDIA vGPU devices on top of Kubernetes
  ☆133Updated last week
• coreweave / tensorizer
  Module, Model, and Tensor Serialization/Deserialization
  ☆241Updated 2 weeks ago
• akx / ollama-dl
  Download models from the Ollama library, without Ollama
  ☆86Updated 7 months ago
• gpustack / vox-box
  A text-to-speech and speech-to-text server compatible with the OpenAI API, supporting Whisper, FunASR, Bark, and CosyVoice backends.
  ☆129Updated 2 weeks ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆76Updated 3 weeks ago
• limcheekin / open-text-embeddings
  Open Source Text Embedding Models with OpenAI Compatible API
  ☆154Updated 11 months ago
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆131Updated last year
• Yard1 / Ray-DeepSpeed-Inference
  ☆17Updated 2 years ago
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆217Updated 6 months ago
• sgl-project / sgl-project.github.io
  This is the documentation repository for SGLang. It is auto-generated from https://github.com/sgl-project/sglang/tree/main/docs.
  ☆53Updated this week
• triton-inference-server / vllm_backend
  ☆268Updated 2 weeks ago
• chaunceyjiang / fake-gpu
  This project is designed to simulate GPU information, making it easier to test scenarios where a GPU is not available.
  ☆47Updated 3 months ago
• NVIDIA / gpu-feature-discovery
  GPU plugin to the node feature discovery for Kubernetes
  ☆300Updated last year
• LiveBench / liveswebench
  ☆42Updated 2 months ago
• bentoml / BentoVLLM
  Self-host LLMs with vLLM and BentoML
  ☆123Updated this week
• wdlctc / headinfer
  ☆50Updated last month
• cloud-gpus / cloud-gpus.github.io
  ☆50Updated this week
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆129Updated last month
• inferx-net / inferx
  InferX is a Inference Function as a Service Platform
  ☆111Updated last week