dipampaul17 / KVSplitLinks

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.
350Updated 2 weeks ago

Alternatives and similar repositories for KVSplit

Users that are interested in KVSplit are comparing it to the libraries listed below

Sorting: