LambdaLabsML / llama
Inference code for LLaMA models
☆41Updated last year
Alternatives and similar repositories for llama:
Users that are interested in llama are comparing it to the libraries listed below
- vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs☆88Updated this week
- ☆116Updated 8 months ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆107Updated last month
- Bamboo-7B Large Language Model☆89Updated 9 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆64Updated 4 months ago
- GPTQ inference Triton kernel☆291Updated last year
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆40Updated last year
- Fast Inference of MoE Models with CPU-GPU Orchestration☆179Updated 2 months ago
- Experiments on speculative sampling with Llama models☆122Updated last year
- Easy and Efficient Quantization for Transformers☆191Updated last month
- An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).☆230Updated 2 months ago
- ☆52Updated 4 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆47Updated this week
- ☆190Updated last month
- QuIP quantization☆48Updated 10 months ago
- Data preparation code for Amber 7B LLM☆84Updated 8 months ago
- Benchmark suite for LLMs from Fireworks.ai☆64Updated last month
- KV cache compression for high-throughput LLM inference☆104Updated last month
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ, and export to onnx/onnx-runtime easily.☆153Updated 3 months ago
- IBM development fork of https://github.com/huggingface/text-generation-inference☆58Updated last month
- scalable and robust tree-based speculative decoding algorithm☆329Updated 5 months ago
- ☆43Updated 6 months ago
- Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".☆263Updated last year
- Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"☆358Updated 10 months ago
- Comparison of Language Model Inference Engines☆202Updated last month
- ☆114Updated 10 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆257Updated 3 months ago
- ☆57Updated 7 months ago
- SparseGPT + GPTQ Compression of LLMs like LLaMa, OPT, Pythia☆41Updated last year
- SAIL: Search Augmented Instruction Learning☆158Updated last year