codefuse-ai / Collinear-Constrained-AttentionLinks
☆63Updated last year
Alternatives and similar repositories for Collinear-Constrained-Attention
Users that are interested in Collinear-Constrained-Attention are comparing it to the libraries listed below
Sorting:
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆78Updated last year
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆123Updated 7 months ago
- Official completion of “Training on the Benchmark Is Not All You Need”.☆35Updated 7 months ago
- A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…☆56Updated 2 years ago
- An Experiment on Dynamic NTK Scaling RoPE☆64Updated last year
- code for Scaling Laws of RoPE-based Extrapolation☆73Updated last year
- ☆108Updated last year
- Code for paper "Patch-Level Training for Large Language Models"☆86Updated 9 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆136Updated last year
- ☆104Updated last month
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper☆33Updated last year
- [ICLR 2025] LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization☆40Updated 5 months ago
- [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models☆52Updated 6 months ago
- [ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation☆113Updated 2 months ago
- Ring is a reasoning MoE LLM provided and open-sourced by InclusionAI, derived from Ling.☆89Updated last week
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆58Updated last year
- ☆36Updated 11 months ago
- [ACL 2024] Long-Context Language Modeling with Parallel Encodings☆157Updated last year
- [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Models☆54Updated 8 months ago
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…☆102Updated last year
- [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623☆86Updated 10 months ago
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales☆32Updated 2 years ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆54Updated 2 years ago
- [ACL 2024] The official codebase for the paper "Self-Distillation Bridges Distribution Gap in Language Model Fine-tuning".☆126Updated 9 months ago
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆61Updated 2 years ago
- Scaling Preference Data Curation via Human-AI Synergy☆95Updated last month
- A unified suite for generating elite reasoning problems and training high-performance LLMs, including pioneering attention-free architect…☆64Updated 2 months ago
- ☆96Updated 3 months ago
- ☆14Updated last year
- [EMNLP 2023]Context Compression for Auto-regressive Transformers with Sentinel Tokens☆25Updated last year