bzhangGo / rmsnorm
Root Mean Square Layer Normalization
☆233Updated 2 years ago
Alternatives and similar repositories for rmsnorm:
Users that are interested in rmsnorm are comparing it to the libraries listed below
- (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …☆159Updated 10 months ago
- ☆141Updated last year
- Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"☆372Updated last year
- Rectified Rotary Position Embeddings☆361Updated 10 months ago
- [ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408☆196Updated last year
- ☆145Updated last year
- Sequence modeling with Mega.☆295Updated 2 years ago
- Code for the ALiBi method for transformer language models (ICLR 2022)☆519Updated last year
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆321Updated 9 months ago
- Implementation of fused cosine similarity attention in the same style as Flash Attention☆212Updated 2 years ago
- Official implementation of TransNormerLLM: A Faster and Better LLM☆243Updated last year
- Implementation of a Transformer, but completely in Triton☆261Updated 2 years ago
- Large Context Attention☆696Updated 2 months ago
- ☆157Updated last year
- Official PyTorch implementation of QA-LoRA☆129Updated last year
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models☆274Updated last month
- ☆102Updated last year
- Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch☆226Updated 6 months ago
- GPTQ inference Triton kernel☆299Updated last year
- Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"☆278Updated 3 weeks ago
- A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models☆720Updated last year
- Prune a model while finetuning or training.☆401Updated 2 years ago
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆60Updated last year
- [KDD'22] Learned Token Pruning for Transformers☆96Updated 2 years ago
- Triton-based implementation of Sparse Mixture of Experts.☆209Updated 4 months ago
- Tutel MoE: An Optimized Mixture-of-Experts Implementation☆789Updated this week
- [ICLR 2022] Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention☆188Updated 2 years ago
- Implementation of Linformer for Pytorch☆276Updated last year
- ☆220Updated 9 months ago
- Get down and dirty with FlashAttention2.0 in pytorch, plug in and play no complex CUDA kernels☆102Updated last year