Code accompanying the paper "Massive Activations in Large Language Models"
☆205Mar 4, 2024Updated 2 years ago
Alternatives and similar repositories for massive-activations
Users that are interested in massive-activations are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)☆164Jul 8, 2025Updated last year
- [ICML 2024] Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrati…☆45Jun 30, 2024Updated 2 years ago
- [COLING'25] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?☆82Jan 22, 2025Updated last year
- Code for studying the super weight in LLM☆124Dec 3, 2024Updated last year
- An algorithm for weight-activation quantization (W4A4, W4A8) of LLMs, supporting both static and dynamic quantization☆178Nov 26, 2025Updated 11 months ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- [EMNLP 2024] RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization☆40Sep 24, 2024Updated 2 years ago
- ☆16Apr 26, 2023Updated 3 years ago
- ☆42Mar 28, 2024Updated 2 years ago
- [NeurIPS 2024 Oral🔥] DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs.☆188Apr 24, 2026Updated 6 months ago
- 🎓Automatically Update circult-eda-mlsys-tinyml Papers Daily using Github Actions (Update Every 8th hours)☆10Updated this week
- ACL24☆11Jun 7, 2024Updated 2 years ago
- [ECCV 2024 Oral] Code for paper: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Langua…☆598Jan 4, 2025Updated last year
- Code for reproducing our paper "Not All Language Model Features Are Linear"☆94Nov 27, 2024Updated last year
- ☆19Jan 3, 2025Updated last year
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- codes and plots for "Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs"☆11Dec 30, 2024Updated last year
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆95Oct 22, 2024Updated 2 years ago
- Official code release for Delta Activations: A Representation for Finetuned Large Language Models☆22Sep 5, 2025Updated last year
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆437Nov 20, 2025Updated 11 months ago
- A simple and effective LLM pruning approach.☆882Aug 9, 2024Updated 2 years ago
- [COLM 2024] SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models☆25Oct 5, 2024Updated 2 years ago
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆544Feb 10, 2025Updated last year
- [ICML 2025] Official PyTorch implementation of "FlatQuant: Flatness Matters for LLM Quantization"☆229Nov 25, 2025Updated 11 months ago
- ☆18May 16, 2026Updated 5 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Relabeling of the POPE benchmark☆14May 16, 2025Updated last year
- ☆173Feb 15, 2025Updated last year
- [NeurIPS 2023] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer☆31Dec 6, 2023Updated 2 years ago
- [ICML'24 Oral] APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference☆49Jun 4, 2024Updated 2 years ago
- Official Pytorch Implementation of "Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity"☆81Jul 7, 2025Updated last year
- [ICML'25] Our study systematically investigates massive values in LLMs' attention mechanisms. First, we observe massive values are concen…☆87Jun 20, 2025Updated last year
- [NeurIPS 2024 Spotlight] Code and data for the paper "Finding Transformer Circuits with Edge Pruning".☆72Aug 15, 2025Updated last year
- ☆22Feb 5, 2024Updated 2 years ago
- [ACL 2025] Are Your LLMs Capable of Stable Reasoning?☆33Aug 5, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- [COLM 2025] DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation; 知乎：https://zhuanlan.zhihu.c…☆30Mar 5, 2025Updated last year
- The Official Implementation of Ada-KV [NeurIPS 2025]☆143Nov 26, 2025Updated 11 months ago
- Code Repository of Evaluating Quantized Large Language Models☆136Sep 8, 2024Updated 2 years ago
- Fast Hadamard transform in CUDA, with a PyTorch interface☆356Mar 10, 2026Updated 7 months ago
- ThinK: Thinner Key Cache by Query-Driven Pruning☆30Jun 2, 2026Updated 4 months ago
- Codebase for the paper "A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language"☆11Oct 11, 2024Updated 2 years ago
- Reorder-based post-training quantization for large language model☆201May 17, 2023Updated 3 years ago