Smart OpenAI‑compatible proxy for llama.cpp: manages slots, saves/restores KV cache to disk, routes requests by prefix similarity, and protects hot slots from being overwritten. Accelerates long prompts (30–60k tokens) via instant reuse or fast on‑demand restore; supports SSE streaming and non‑stream JSON over /v1/chat/completions.
☆56Sep 16, 2026Updated last month
Alternatives and similar repositories for proxycache
Users that are interested in proxycache are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- LLM backed Fantasy Tribe Game☆19Nov 21, 2024Updated last year
- Tuning and qualification harness for the NVIDIA CMP 170HX (GA100). Measures, gates and recovers clock/voltage settings, and treats a comp…☆89Sep 28, 2026Updated 2 weeks ago
- BROKEN REPO. DO NOT USE UNDER ANY CIRCUMSTANCES☆20Updated this week
- LLM (v)RAM estimator for GGUF models from huggingface and ollama across various quantisation and context sizes☆13Nov 26, 2025Updated 10 months ago
- Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.☆158Updated this week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Generate a llama-quantize command to copy the quantization parameters of any GGUF☆36Apr 20, 2026Updated 5 months ago
- Agentic BYOK Browser-Based Website Builder☆64Oct 11, 2026Updated last week
- llama.cpp fork with additional SOTA quants and improved performance☆22Oct 4, 2026Updated 2 weeks ago
- ☆59Oct 10, 2025Updated last year
- Turbo Lossless - 1.33x Smaller, 2.93x Faster, Decode with 1 ADD operation☆15Apr 4, 2026Updated 6 months ago
- Proxy for OpenAI☆16Sep 2, 2025Updated last year
- Private-first, self-hostable knowledge base. Your data, your server, your control. No cloud, no telemetry, no trust required.☆22Oct 4, 2026Updated 2 weeks ago
- CLI secret management☆16Sep 24, 2026Updated 3 weeks ago
- A lightweight, standalone Go microservice for streaming ZIP archives on-the-fly from S3-compatible storage or local filesystems.☆13Nov 30, 2025Updated 10 months ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- bandwidth limiting middleware plugin for Traefik that provides fine-grained control over data transfer rates. This plugin supports per-ba…☆16Apr 20, 2026Updated 5 months ago
- High-performance batched Top-K selection for CPU inference. Up to 80x faster than PyTorch, optimized for LLM sampling with AVX2 SIMD.☆18Mar 20, 2026Updated 6 months ago
- Branch-Thinking MCP Tool A TypeScript-powered MCP server for managing parallel branches of thought, semantic cross-references, and persis…☆15Apr 25, 2025Updated last year
- Enhancing LLMs with LoRA☆228Oct 20, 2025Updated 11 months ago
- Terminal database client with AI chat and an MCP server☆105Oct 10, 2026Updated last week
- SR²AM: Efficient Agentic Reasoning Through Self-Regulated Simulative Planning☆21May 22, 2026Updated 4 months ago
- A Kubernetes controller and webhook implementation that enables safe, staged rollouts of DaemonSets☆16Jul 31, 2025Updated last year
- Yet Another (LLM) Web UI, made with Gemini☆12Dec 25, 2024Updated last year
- ☆18Jan 5, 2016Updated 10 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- llama.cpp fork with additional SOTA quants and improved performance☆3,303Updated this week
- VellumForge2 is a Golang CLI for generating high-quality Direct Preference Optimization datasets via a hierarchical prompt pipeline with …☆22Feb 15, 2026Updated 8 months ago
- Plugin for Octoprint that displays what layer a print job is on.☆18Aug 4, 2021Updated 5 years ago
- Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc☆5,990Updated this week
- Deploying full-stack on-prem deep research agent that can be run entirely on a local machine for $0!☆34Nov 8, 2025Updated 11 months ago
- ☆29May 12, 2026Updated 5 months ago
- ☆45Oct 9, 2025Updated last year
- A zero-allocation, header-only C++ BPE tokenizer for Qwen, built for maximum inference throughput.☆24Apr 3, 2026Updated 6 months ago
- 🎙️ VibeVoice FastAPI - Multi-Speaker TTS API☆33Aug 27, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆34Apr 12, 2026Updated 6 months ago
- A high performance authentication and access-control gateway for LLM API backends☆17Jun 11, 2026Updated 4 months ago
- FamilyBench evaluation tool for testing the relational reasoning capabilities of Large Language Models (LLMs).☆46Sep 3, 2026Updated last month
- ☆17Nov 28, 2018Updated 7 years ago
- btop-style terminal dashboards for Prometheus & Grafana.☆54Oct 5, 2026Updated last week
- HIP/ROCm fork optimized for AMD RDNA2 (gfx1030) with PrismML Q1_0_G128 1-bit quant support, RotorQuant, TurboQuant, EAGLE3 and P-EAGLE sp…☆26Jun 23, 2026Updated 3 months ago
- For a number of years now, work has been proceeding in order to bring to perfection the crudely-conceived idea of a machine that would no…☆14Nov 12, 2025Updated 11 months ago