LLM Inference on consumer devices
☆131Mar 17, 2025Updated last year
Alternatives and similar repositories for UMbreLLa
Users that are interested in UMbreLLa are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆71Nov 4, 2024Updated last year
- scalable and robust tree-based speculative decoding algorithm☆375Jan 28, 2025Updated last year
- ☆27Jun 11, 2025Updated last year
- [ICLR2025 Spotlight] MagicPIG: LSH Sampling for Efficient LLM Generation☆256Dec 16, 2024Updated last year
- Vortex: Programmable Sparse Attention for Agents as Algorithm Designers☆69Aug 22, 2026Updated 2 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- FusionQuant Model Merge & GGUF Conversion Pipeline - Your Free Toolkit for Custom LLMs!☆14Jun 24, 2025Updated last year
- [ACL 2026 (Main)] LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification☆86Jul 14, 2025Updated last year
- Compute Where It Counts: Scaling both speed and quality with training compute☆13Sep 18, 2025Updated last year
- Bloat Free, Portable and Lightweight LLM Frontend (Single HTML file). With Lorebook, Web Search, Macro Engine etc.☆23Oct 25, 2026Updated last week
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆90Sep 15, 2025Updated last year
- ☆17Jun 15, 2026Updated 4 months ago
- Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)☆415Apr 22, 2025Updated last year
- Work in progress.☆82Nov 25, 2025Updated 11 months ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆173Dec 23, 2025Updated 10 months ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- ☆16Dec 16, 2024Updated last year
- ☆24Jan 22, 2025Updated last year
- Collaborative inference of latent diffusion via hivemind☆12May 29, 2023Updated 3 years ago
- High-Throughput Batch Inference☆18Updated this week
- A local-first LLM development studio. Build, test, and customize inference workflows with your own models — no cloud, totally local.☆16May 21, 2025Updated last year
- A Blueprint-style visual node editor for creating FastMCP servers. Build MCP tools, resources, and prompts by connecting nodes - no codin…☆26Dec 8, 2025Updated 10 months ago
- a simple API to use CUPTI☆10Aug 19, 2025Updated last year
- Sirius, an efficient correction mechanism, which significantly boosts Contextual Sparsity models on reasoning tasks while maintaining its…☆22Sep 10, 2024Updated 2 years ago
- ☆79Nov 26, 2024Updated last year
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- GUI tool to QLoRA/LoRA-fine-tune LLMs and deploy to Ollama. Broad GPU support (NVIDIA/AMD/Intel/Apple) + CPU fallback.☆15Feb 18, 2026Updated 8 months ago
- Fast Hadamard transform in CUDA, with a PyTorch interface☆356Mar 10, 2026Updated 7 months ago
- Validate database backup files before migration to prevent costly failures☆14Aug 4, 2025Updated last year
- [COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding☆280Aug 31, 2024Updated 2 years ago
- Run BitNet b1.58 ternary LLMs with WebGPU — in browsers and native apps☆23Oct 23, 2026Updated last week
- ☆20Nov 26, 2025Updated 11 months ago
- Cognito: Supercharge your Chrome browser with AI. Guide, query, and control everything using natural language.☆57Updated this week
- Simple python library for generating your own perfetto traces for your application. Can be used for both app instrumentation and custom …☆27Jun 22, 2025Updated last year
- Official Implementation of SAM-Decoding: Speculative Decoding via Suffix Automaton☆54May 12, 2026Updated 5 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆350Jul 2, 2024Updated 2 years ago
- ☆56Nov 5, 2024Updated last year
- A forward proxy to turn network traffic into personal memory for AI agents☆39Mar 30, 2026Updated 7 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆186Jul 12, 2024Updated 2 years ago
- Glyphs, acting as collaboratively defined symbols linking related concepts, add a layer of multidimensional semantic richness to user-AI …☆57Feb 10, 2025Updated last year
- 🚀 FlexLLama - Lightweight self-hosted tool for running multiple llama.cpp server instances with OpenAI v1 API compatibility and multi-GP…☆62Sep 26, 2026Updated last month
- The SchemaPin protocol for cryptographically signing and verifying AI agent tool schemas to prevent supply-chain attacks.☆16Oct 11, 2026Updated 3 weeks ago