ducdauge / sft-llmLinks
Scaling Sparse Fine-Tuning to Large Language Models
☆16Updated last year
Alternatives and similar repositories for sft-llm
Users that are interested in sft-llm are comparing it to the libraries listed below
Sorting:
- The open-source materials for paper "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity".☆23Updated 7 months ago
- ☆18Updated 11 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆52Updated 2 years ago
- ☆25Updated last year
- Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"☆28Updated last year
- ☆20Updated last year
- The official implementation for Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free☆44Updated last month
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- ☆55Updated 11 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆28Updated 9 months ago
- Using FlexAttention to compute attention with different masking patterns☆44Updated 9 months ago
- Xmixers: A collection of SOTA efficient token/channel mixers☆11Updated 7 months ago
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆37Updated last year
- ☆21Updated last year
- ☆19Updated 2 years ago
- triton ver of gqa flash attn, based on the tutorial☆11Updated 10 months ago
- Codebase for Instruction Following without Instruction Tuning☆34Updated 9 months ago
- [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models☆52Updated 4 months ago
- Implementation of the model: "Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models" in PyTorch☆30Updated 3 weeks ago
- [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models☆31Updated last year
- ☆48Updated last year
- Official code repo for paper "Great Memory, Shallow Reasoning: Limits of kNN-LMs"☆23Updated last month
- ☆32Updated last year
- Code for "RSQ: Learning from Important Tokens Leads to Better Quantized LLMs"☆18Updated last week
- Here we will test various linear attention designs.☆59Updated last year
- [ACL'24 Oral] Analysing The Impact of Sequence Composition on Language Model Pre-Training☆22Updated 10 months ago
- ☆12Updated last year
- ☆32Updated last year
- Official implementation of the paper: "A deeper look at depth pruning of LLMs"☆15Updated 11 months ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆42Updated last year