zyushun / hessian-spectrum

Related projects ⓘ

Alternatives and complementary repositories for hessian-spectrum

• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆34Updated last week
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆123Updated 8 months ago
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆79Updated last year
• tml-epfl / why-weight-decay
  Why Do We Need Weight Decay in Modern Deep Learning? [NeurIPS 2024]
  ☆52Updated last month
• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆24Updated 2 weeks ago
• berlino / seq_icl
  ☆50Updated 6 months ago
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆28Updated 7 months ago
• locuslab / edge-of-stability
  ☆59Updated 3 years ago
• JonasGeiping / linear_cross_entropy_loss
  A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.
  ☆54Updated 3 months ago
• sustcsonglin / mamba-triton
  ☆45Updated 9 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆44Updated last year
• teelinsan / parallel-decoding
  Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"
  ☆109Updated 8 months ago
• epfml / dynamic-sparse-flash-attention
  ☆132Updated last year
• sail-sg / Attention-Sink
  [ATTRIB @ NeurIPS 2024] When Attention Sink Emerges in Language Models: An Empirical View
  ☆29Updated last month
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆24Updated 7 months ago
• wjxts / RegularizedBN
  ☆21Updated last year
• insuhan / hyper-attn
  ☆74Updated 11 months ago
• gregorbachmann / Next-Token-Failures
  ☆69Updated 8 months ago
• lzhangbv / eva
  [ICLR 2023] Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation
  ☆12Updated last year
• microsoft / SparseMixer
  Sparse Backpropagation for Mixture-of-Expert Training
  ☆24Updated 4 months ago
• automl / is_mamba_capable_of_icl
  ☆13Updated 6 months ago
• goombalab / phi-mamba
  Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…
  ☆78Updated 2 months ago
• ZO-Bench / ZO-LLM
  [ICML 2024] Official code for the paper "Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark ".
  ☆73Updated 4 months ago
• ryoungj / ObsScaling
  [NeurIPS'24 Spotlight] Observational Scaling Laws
  ☆44Updated last month
• berlino / gated_linear_attention
  ☆98Updated 8 months ago
• mengxiayu / LLMSuperWeight
  Code for studying the super weight in LLM
  ☆16Updated last week
• sjelassi / transformers_ssm_copy
  ☆24Updated 8 months ago
• epfml / schedules-and-scaling
  ☆53Updated 3 weeks ago
• Leiay / looped_transformer
  ☆20Updated 11 months ago
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆49Updated 3 weeks ago