Alternatives and similar repositories for ggml

Users that are interested in ggml are comparing it to the libraries listed below

• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆87Updated this week
• nomic-ai / kompute
  General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). …
  ☆51Updated 4 months ago
• leloykun / llama2.cpp
  Inference Llama 2 in one file of pure C++
  ☆83Updated last year
• abetlen / ggml-python
  Python bindings for ggml
  ☆142Updated 10 months ago
• BlinkDL / fast.c
  Prepare for DeekSeek R1 inference: Benchmark CPU, DRAM, SSD, iGPU, GPU, ... with efficient code.
  ☆72Updated 5 months ago
• hscspring / llama.np
  Inference Llama/Llama2/Llama3 Modes in NumPy
  ☆21Updated last year
• catid / bitnet_cpu
  Experiments with BitNet inference on CPU
  ☆54Updated last year
• anarchy-ai / llm-speed-benchmark
  Benchmarking tool for assessing LLM models' performance across different hardwares
  ☆17Updated last year
• furiousteabag / vram-calculator
  Transformer GPU VRAM estimator
  ☆66Updated last year
• unslothai / llama.cpp
  LLM inference in C/C++
  ☆94Updated this week
• xaedes / llama.cpp
  Port of Facebook's LLaMA model in C/C++
  ☆22Updated last year
• okuvshynov / cubestat
  Horizon chart for CPU/GPU/Neural Engine utilization monitoring. Supports Apple M1-M4, Nvidia GPUs, AMD GPUs
  ☆25Updated 3 weeks ago
• facebookresearch / fastgen
  Simple high-throughput inference library
  ☆120Updated 2 months ago
• IST-DASLab / qmoe
  Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".
  ☆277Updated last year
• antirez / LLM-FTC-sampling
  First token cutoff sampling inference example
  ☆30Updated last year
• huggingface / optimum-amd
  AMD related optimizations for transformer models
  ☆80Updated 3 weeks ago
• xyzhang626 / embeddings.cpp
  ggml implementation of embedding models including SentenceTransformer and BGE
  ☆58Updated last year
• lachlansneff / sparsellama
  ☆40Updated 2 years ago
• wangcx18 / llm-vscode-inference-server
  An endpoint server for efficiently serving quantized open-source LLMs for code.
  ☆55Updated last year
• AlpinDale / QuIP-for-Llama
  Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees" adapted for Llama models
  ☆38Updated last year
• NolanoOrg / SpectraSuite
  ☆49Updated 11 months ago
• vikhyat / mixtral-inference
  inference code for mixtral-8x7b-32kseqlen
  ☆100Updated last year
• petals-infra / health.petals.dev
  🏥 Health monitor for a Petals swarm
  ☆39Updated 11 months ago
• abhisheknair10 / llama3.cu
  Lightweight Llama 3 8B Inference Engine in CUDA C
  ☆47Updated 3 months ago
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆264Updated 9 months ago
• RobinQu / instinct.cpp
  instinct.cpp provides ready to use alternatives to OpenAI Assistant API and built-in utilities for developing AI Agent applications (RAG,…
  ☆52Updated last year
• vcskaushik / LLMzip
  ☆57Updated 6 months ago
• casper-hansen / AutoAWQ_kernels
  ☆75Updated 7 months ago
• exo-explore / mlx-bitnet
  1.58 Bit LLM on Apple Silicon using MLX
  ☆214Updated last year
• kroggen / mamba.c
  Inference of Mamba models in pure C
  ☆188Updated last year