Alternatives and similar repositories for GPU-Benchmarks-on-LLM-Inference

Users that are interested in GPU-Benchmarks-on-LLM-Inference are comparing it to the libraries listed below

• aphrodite-engine / aphrodite-engine
  Large-scale LLM inference engine
  ☆1,577Updated 2 weeks ago
• e-p-armstrong / augmentoolkit
  Create Custom LLMs
  ☆1,754Updated last month
• turboderp-org / exllamav2
  A fast inference library for running LLMs locally on modern consumer-class GPUs
  ☆4,346Updated 2 months ago
• Blaizzy / mlx-vlm
  MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
  ☆1,748Updated this week
• tinygrad / open-gpu-kernel-modules
  NVIDIA Linux open GPU with P2P support
  ☆1,263Updated 4 months ago
• codelion / optillm
  Optimizing inference proxy for LLMs
  ☆3,042Updated last week
• NVIDIA / RULER
  This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
  ☆1,327Updated 2 weeks ago
• b4rtaz / distributed-llama
  Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
  ☆2,705Updated 2 weeks ago
• mostlygeek / llama-swap
  Reliable model swapping for any local OpenAI compatible server - llama.cpp, vllm, etc
  ☆1,730Updated this week
• theroyallab / tabbyAPI
  The official API server for Exllama. OAI compatible, lightweight, and fast.
  ☆1,068Updated last week
• RahulSChand / gpu_poor
  Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
  ☆1,378Updated 10 months ago
• noamgat / lm-format-enforcer
  Enforce the output format (JSON Schema, Regex etc) of a language model
  ☆1,942Updated 2 months ago
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆1,032Updated 10 months ago
• Maximilian-Winter / llama-cpp-agent
  The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM …
  ☆601Updated 8 months ago
• NousResearch / Hermes-Function-Calling
  ☆1,103Updated last year
• michaelfeil / infinity
  Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
  ☆2,513Updated 2 weeks ago
• ikawrakow / ik_llama.cpp
  llama.cpp fork with additional SOTA quants and improved performance
  ☆1,266Updated this week
• casper-hansen / AutoAWQ
  AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:
  ☆2,260Updated 5 months ago
• vllm-project / llm-compressor
  Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
  ☆2,106Updated last week
• av / harbor
  Effortlessly run LLM backends, APIs, frontends, and services with one command.
  ☆2,117Updated 3 weeks ago
• lmstudio-ai / mlx-engine
  LM Studio Apple MLX engine
  ☆799Updated this week
• foldl / chatllm.cpp
  Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
  ☆721Updated last week
• huggingface / smollm
  Everything about the SmolLM and SmolVLM family of models
  ☆3,332Updated last month
• jonfairbanks / local-rag
  Ingest files for retrieval augmented generation (RAG) with open-source Large Language Models (LLMs), all without 3rd parties or sensitive…
  ☆706Updated last year
• microsoft / VPTQ
  VPTQ, A Flexible and Extreme low-bit quantization algorithm
  ☆659Updated 6 months ago
• turboderp / exllama
  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
  ☆2,903Updated 2 years ago
• SomeOddCodeGuy / WilmerAI
  WilmerAI is one of the oldest LLM semantic routers. It uses multi-layer prompt routing and complex workflows to allow you to not only cre…
  ☆784Updated last week
• vince-lam / awesome-local-llms
  Compare open-source local LLM inference projects by their metrics to assess popularity and activeness.
  ☆664Updated last month
• dCaples / AutoDidact
  Autonomously train research-agent LLMs on custom data using reinforcement learning and self-verification.
  ☆667Updated 7 months ago
• stanford-oval / WikiChat
  WikiChat is an improved RAG. It stops the hallucination of large language models by retrieving data from a corpus.
  ☆1,525Updated 5 months ago