togethercomputer / stripedhyena
Repository for StripedHyena, a state-of-the-art beyond Transformer architecture
☆253Updated 6 months ago
Related projects: ⓘ
- ☆169Updated this week
- Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"☆530Updated 4 months ago
- Multipack distributed sampler for fast padding-free training of LLMs☆170Updated last month
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆206Updated last month
- Bi-Directional Equivariant Long-Range DNA Sequence Modeling☆144Updated last month
- Understand and test language model architectures on synthetic tasks.☆156Updated 4 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆94Updated 2 weeks ago
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆452Updated last month
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆271Updated 4 months ago
- (Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens" (https://arxiv.org/abs/2307…☆49Updated last year
- Code repository for Black Mamba☆218Updated 7 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆155Updated 8 months ago
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆278Updated 3 months ago
- X-LoRA: Mixture of LoRA Experts☆152Updated last month
- Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".☆258Updated 10 months ago
- Official implementation for HyenaDNA, a long-range genomic foundation model built with Hyena☆567Updated 3 months ago
- batched loras☆327Updated last year
- The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction☆361Updated 2 months ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆155Updated 2 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆84Updated 4 months ago
- ☆190Updated last week
- Implementation of DoRA☆278Updated 3 months ago
- Recurrent Memory Transformer☆148Updated last year
- Website for hosting the Open Foundation Models Cheat Sheet.☆257Updated 2 months ago
- Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch☆222Updated 2 weeks ago
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆248Updated 10 months ago
- ☆174Updated 4 months ago
- Simplified Masked Diffusion Language Model☆160Updated 2 weeks ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆337Updated last month
- ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward exp…☆218Updated 5 months ago