codefuse-ai / Collinear-Constrained-AttentionLinks
☆63Updated last year
Alternatives and similar repositories for Collinear-Constrained-Attention
Users that are interested in Collinear-Constrained-Attention are comparing it to the libraries listed below
Sorting:
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆123Updated 7 months ago
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆78Updated last year
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…☆102Updated last year
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆58Updated last year
- Inference Code for Paper "Harder Tasks Need More Experts: Dynamic Routing in MoE Models"☆62Updated last year
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales☆32Updated 2 years ago
- ☆110Updated last year
- Ring is a reasoning MoE LLM provided and open-sourced by InclusionAI, derived from Ling.☆95Updated last month
- Linear Attention Sequence Parallelism (LASP)☆86Updated last year
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper☆33Updated last year
- ☆14Updated last year
- An Experiment on Dynamic NTK Scaling RoPE☆64Updated last year
- code for Scaling Laws of RoPE-based Extrapolation☆73Updated last year
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆136Updated last year
- [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models☆54Updated 6 months ago
- Code for paper "Patch-Level Training for Large Language Models"☆86Updated 9 months ago
- [ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation☆113Updated 3 months ago
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejection☆51Updated 10 months ago
- Official implementation of TransNormerLLM: A Faster and Better LLM☆247Updated last year
- Official completion of “Training on the Benchmark Is Not All You Need”.☆35Updated 8 months ago
- A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…☆56Updated 2 years ago
- [ACL 2024] Long-Context Language Modeling with Parallel Encodings☆157Updated last year
- Code for ICML 25 paper "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"☆42Updated 2 months ago
- Counting-Stars (★)☆83Updated 3 months ago
- ☆104Updated last month
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆54Updated 2 years ago
- ☆13Updated 11 months ago
- Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"☆149Updated 5 months ago
- ☆48Updated 10 months ago
- [ACL 24 Findings] Implementation of Resonance RoPE and the PosGen synthetic dataset.☆23Updated last year