IST-DASLab / marlinLinks

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
831Updated 8 months ago

Alternatives and similar repositories for marlin

Users that are interested in marlin are comparing it to the libraries listed below

Sorting: