Alternatives and similar repositories for llama.cpp

Users that are interested in llama.cpp are comparing it to the libraries listed below

• mlc-ai / relax
  ☆172Updated this week
• mlc-ai / llm-perf-bench
  ☆120Updated last year
• marty1885 / llama.cpp
  My develoopment fork of llama.cpp. For now working on RK3588 NPU and Tenstorrent backend
  ☆114Updated 2 weeks ago
• Cornell-RelaxML / quip-sharp
  ☆577Updated last year
• tenstorrent / tt-buda-demos
  Repository of model demos using TT-Buda
  ☆63Updated 10 months ago
• microsoft / BitBLAS
  BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment.
  ☆749Updated 6 months ago
• moonshine-ai / useful-transformers
  Efficient Inference of Transformer models
  ☆478Updated last year
• rejunity / tiny-asic-1_58bit-matrix-mul
  Tiny ASIC implementation for "The Era of 1-bit LLMs All Large Language Models are in 1.58 Bits" matrix multiplication unit
  ☆175Updated last year
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆93Updated last week
• intel / intel-xpu-backend-for-triton
  OpenAI Triton backend for Intel® GPUs
  ☆226Updated this week
• fpgaminer / GPTQ-triton
  GPTQ inference Triton kernel
  ☆321Updated 2 years ago
• ROCm / triton
  Development repository for the Triton language and compiler
  ☆140Updated last week
• tpoisonooo / llama.onnx
  LLaMa/RWKV onnx models, quantization and testcase
  ☆367Updated 2 years ago
• KevlarKanou / rwkv7.c
  Inference RWKV v7 in pure C.
  ☆44Updated 3 months ago
• IST-DASLab / marlin
  FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
  ☆1,005Updated last year
• Aaronhuang-778 / BiLLM
  [ICML 2024] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
  ☆229Updated last year
• tenstorrent / tt-buda
  Tenstorrent TT-BUDA Repository
  ☆314Updated 10 months ago
• mtx512 / rk3588-npu
  Reverse engineering the rk3588 npu
  ☆110Updated last year
• BlinkDL / fast.c
  Prepare for DeekSeek R1 inference: Benchmark CPU, DRAM, SSD, iGPU, GPU, ... with efficient code.
  ☆74Updated last year
• microsoft / VPTQ
  VPTQ, A Flexible and Extreme low-bit quantization algorithm
  ☆674Updated 9 months ago
• IST-DASLab / qmoe
  Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".
  ☆280Updated 2 years ago
• Cornell-RelaxML / QuIP
  Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"
  ☆395Updated last year
• wangsiping97 / FastGEMV
  High-speed GEMV kernels, at most 2.7x speedup compared to pytorch baseline.
  ☆127Updated last year
• intel / neural-speed
  An innovative library for efficient LLM inference via low-bit quantization
  ☆352Updated last year
• NolanoOrg / cformers
  SoTA Transformers with C-backend for fast inference on your CPU.
  ☆311Updated 2 years ago
• anthonix / llm.c
  LLM training in simple, raw C/HIP for AMD GPUs
  ☆58Updated last year
• Xilinx / llvm-aie
  Fork of LLVM to support AMD AIEngine processors
  ☆187Updated this week
• SqueezeAILab / SqueezeLLM
  [ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization
  ☆713Updated last year
• triton-lang / triton-cpu
  An experimental CPU backend for Triton
  ☆174Updated 2 months ago
• usyd-fsalab / fp6_llm
  An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
  ☆276Updated 6 months ago