Code accompanying the paper "Massive Activations in Large Language Models"
☆205Mar 4, 2024Updated 2 years ago
Alternatives and similar repositories for massive-activations
Users that are interested in massive-activations are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)☆164Jul 8, 2025Updated last year
- [ICML 2024] Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrati…☆45Jun 30, 2024Updated 2 years ago
- [COLING'25] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?☆82Jan 22, 2025Updated last year
- Code for studying the super weight in LLM☆124Dec 3, 2024Updated last year
- An algorithm for weight-activation quantization (W4A4, W4A8) of LLMs, supporting both static and dynamic quantization☆178Nov 26, 2025Updated 11 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- [EMNLP 2024] RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization☆40Sep 24, 2024Updated 2 years ago
- ☆16Apr 26, 2023Updated 3 years ago
- 🎓Automatically Update circult-eda-mlsys-tinyml Papers Daily using Github Actions (Update Every 8th hours)☆10Updated this week
- ☆43Mar 28, 2024Updated 2 years ago
- [ICLR 2025] Systematic Outliers in Large Language Models.☆11Feb 11, 2025Updated last year
- ACL24☆11Jun 7, 2024Updated 2 years ago
- ☆10Apr 16, 2024Updated 2 years ago
- [NeurIPS 2024 Oral🔥] DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs.☆190Apr 24, 2026Updated 6 months ago
- ☆19Jan 3, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- (ICLR 2026) Unveiling Super Experts in Mixture-of-Experts Large Language Models☆44Sep 25, 2025Updated last year
- [ECCV 2024 Oral] Code for paper: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Langua…☆602Jan 4, 2025Updated last year
- Code for reproducing our paper "Not All Language Model Features Are Linear"☆94Nov 27, 2024Updated last year
- codes and plots for "Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs"☆11Dec 30, 2024Updated last year
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆95Oct 22, 2024Updated 2 years ago
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆545Feb 10, 2025Updated last year
- [ICML 2025] Official PyTorch implementation of "FlatQuant: Flatness Matters for LLM Quantization"☆230Nov 25, 2025Updated 11 months ago
- A simple and effective LLM pruning approach.☆882Aug 9, 2024Updated 2 years ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆441Nov 20, 2025Updated 11 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- [COLM 2024] SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models☆25Oct 5, 2024Updated 2 years ago
- [NeurIPS 2023] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer☆31Dec 6, 2023Updated 2 years ago
- [ICML'24 Oral] APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference☆49Jun 4, 2024Updated 2 years ago
- Official code release for Delta Activations: A Representation for Finetuned Large Language Models☆22Sep 5, 2025Updated last year
- [ACM MM25] MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models☆14Aug 8, 2025Updated last year
- Official Pytorch Implementation of "Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity"☆81Jul 7, 2025Updated last year
- [ICML'25] Our study systematically investigates massive values in LLMs' attention mechanisms. First, we observe massive values are concen…☆87Jun 20, 2025Updated last year
- ☆173Feb 15, 2025Updated last year
- ☆22Feb 5, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Relabeling of the POPE benchmark☆14May 16, 2025Updated last year
- [COLM 2025] DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation; 知乎:https://zhuanlan.zhihu.c…☆30Mar 5, 2025Updated last year
- The Official Implementation of Ada-KV [NeurIPS 2025]☆143Nov 26, 2025Updated 11 months ago
- ThinK: Thinner Key Cache by Query-Driven Pruning☆30Jun 2, 2026Updated 5 months ago
- ☆19May 16, 2026Updated 6 months ago
- [NeurIPS 2024 Spotlight] Code and data for the paper "Finding Transformer Circuits with Edge Pruning".☆71Aug 15, 2025Updated last year
- Reorder-based post-training quantization for large language model☆201May 17, 2023Updated 3 years ago
Are these results useful?