swiss-ai / MoELinks
some mixture of experts architecture implementations
☆21Updated last year
Alternatives and similar repositories for MoE
Users that are interested in MoE are comparing it to the libraries listed below
Sorting:
- The evaluation framework for training-free sparse attention in LLMs☆96Updated 3 months ago
- Awesome Triton Resources☆33Updated 5 months ago
- Implementation of Hyena Hierarchy in JAX☆10Updated 2 years ago
- Official code for the paper "Attention as a Hypernetwork"☆42Updated last year
- Here we will test various linear attention designs.☆62Updated last year
- Code for the EMNLP24 paper "A simple and effective L2 norm based method for KV Cache compression."☆16Updated 9 months ago
- A repository for research on medium sized language models.☆77Updated last year
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More☆34Updated 4 months ago
- HGRN2: Gated Linear RNNs with State Expansion☆54Updated last year
- Linear Attention Sequence Parallelism (LASP)☆86Updated last year
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated 11 months ago
- Minimum Description Length probing for neural network representations☆18Updated 7 months ago
- Fork of Flame repo for training of some new stuff in development☆17Updated 3 weeks ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆84Updated 10 months ago
- Fast and memory-efficient exact attention☆69Updated 6 months ago
- ☆20Updated last year
- ☆23Updated 5 months ago
- Official implementation of "Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning"☆22Updated 4 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆129Updated 9 months ago
- Intel Gaudi's Megatron DeepSpeed Large Language Models for training☆13Updated 9 months ago
- ☆19Updated 6 months ago
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024) + Arithmetic Transfor…☆11Updated 5 months ago
- RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's l…☆47Updated 2 months ago
- ☆13Updated 5 months ago
- GoldFinch and other hybrid transformer components☆45Updated last year
- Code and data for paper "(How) do Language Models Track State?"☆18Updated 5 months ago
- ☆85Updated last year
- ☆13Updated 6 months ago
- QuIP quantization☆60Updated last year
- ☆54Updated 3 months ago