MingyuJ666 / Rope_with_LLMLinks

[ICML'25] Our study systematically investigates massive values in LLMs' attention mechanisms. First, we observe massive values are concentrated in low-frequency dimensions across different attention heads exclusively in attention queries (Q) and keys (K) while absent in values (V).
66Updated last week

Alternatives and similar repositories for Rope_with_LLM

Users that are interested in Rope_with_LLM are comparing it to the libraries listed below

Sorting: