Alternatives and similar repositories for distributed-llama:

Users that are interested in distributed-llama are comparing it to the libraries listed below

• turboderp-org / exllamav2
  A fast inference library for running LLMs locally on modern consumer-class GPUs
  ☆4,125Updated this week
• distantmagic / paddler
  Stateful load balancer custom-tailored for llama.cpp 🏓🦙
  ☆742Updated 2 weeks ago
• foldl / chatllm.cpp
  Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
  ☆573Updated this week
• EricLBuehler / mistral.rs
  Blazingly fast LLM inference.
  ☆5,437Updated this week
• menloresearch / cortex.cpp
  Local AI API Platform
  ☆2,622Updated last week
• aphrodite-engine / aphrodite-engine
  Large-scale LLM inference engine
  ☆1,384Updated this week
• predibase / lorax
  Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
  ☆2,955Updated this week
• lyogavin / airllm
  AirLLM 70B inference with single 4GB GPU
  ☆5,758Updated 4 months ago
• arcee-ai / mergekit
  Tools for merging pretrained large language models.
  ☆5,571Updated this week
• menloresearch / ichigo
  Local realtime voice AI
  ☆2,279Updated last month
• axolotl-ai-cloud / axolotl
  Go ahead and axolotl questions
  ☆9,137Updated this week
• evilsocket / cake
  Distributed LLM and StableDiffusion inference for mobile, desktop and server.
  ☆2,838Updated 6 months ago
• AnswerDotAI / fsdp_qlora
  Training LLMs with QLoRA + FSDP
  ☆1,470Updated 5 months ago
• Maximilian-Winter / llama-cpp-agent
  The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM …
  ☆551Updated 2 months ago
• NousResearch / DisTrO
  Distributed Training Over-The-Internet
  ☆901Updated 4 months ago
• pytorch / executorch
  On-device AI across mobile, embedded and edge for PyTorch
  ☆2,747Updated this week
• guinmoon / LLMFarm
  llama and other large language models on iOS and MacOS offline using GGML library.
  ☆1,734Updated last month
• lm-sys / RouteLLM
  A framework for serving and evaluating LLM routers - save LLM costs without compromising quality
  ☆3,817Updated 8 months ago
• ictnlp / LLaMA-Omni
  LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve spee…
  ☆2,889Updated 5 months ago
• turboderp / exllama
  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
  ☆2,863Updated last year
• tinygrad / open-gpu-kernel-modules
  NVIDIA Linux open GPU with P2P support
  ☆1,094Updated 4 months ago
• intel / intel-extension-for-transformers
  ⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Pl…
  ☆2,169Updated 6 months ago
• vllm-project / llm-compressor
  Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
  ☆1,237Updated this week
• facebookresearch / MobileLLM
  MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.
  ☆1,288Updated this week
• RahulSChand / gpu_poor
  Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
  ☆1,285Updated 4 months ago
• AutoGPTQ / AutoGPTQ
  An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
  ☆4,818Updated last week
• abgulati / LARS
  An application for running LLMs locally on your device, with your documents, facilitating detailed citations in generated responses.
  ☆579Updated 5 months ago
• marella / ctransformers
  Python bindings for the Transformer models implemented in C/C++ using GGML library.
  ☆1,857Updated last year
• mistralai / mistral-finetune
  ☆2,915Updated 7 months ago
• ParisNeo / ollama_proxy_server
  A proxy server for multiple ollama instances with Key security
  ☆392Updated last week