zyushun / hessian-spectrum

Related projects: ⓘ

• ZO-Bench / ZO-LLM
  [ICML 2024] Official code for the paper "Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark ".
  ☆62Updated 2 months ago
• hendrydong / NTK-and-MF-examples
  Visualization of mean field and neural tangent kernel regime
  ☆20Updated last month
• VITA-Group / SMC-Bench
  [ICLR 2023] "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!" Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen…
  ☆27Updated last year
• allenbai01 / transformers-as-statisticians
  ☆25Updated last year
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆77Updated last year
• OPTML-Group / DeepZero
  [ICLR'24] "DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training" by Aochuan Chen*, Yimeng Zhang*, Jinghan Jia, James Di…
  ☆35Updated 6 months ago
• berlino / gated_linear_attention
  ☆94Updated 6 months ago
• teelinsan / parallel-decoding
  Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"
  ☆99Updated 6 months ago
• Noahs-ARK / RFA
  ☆32Updated 3 years ago
• lzhangbv / eva
  [ICLR 2023] Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation
  ☆10Updated last year
• tml-epfl / why-weight-decay
  Why Do We Need Weight Decay in Modern Deep Learning? [arXiv, Oct 2023]
  ☆41Updated 11 months ago
• gregorbachmann / Next-Token-Failures
  ☆61Updated 6 months ago
• locuslab / edge-of-stability
  ☆56Updated 3 years ago
• VITA-Group / Junk_DNA_Hypothesis
  [ICML 2024] Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity; Lu Yin*, Ajay Jaiswal*, Shiwei Liu, So…
  ☆14Updated 3 months ago
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆25Updated 5 months ago
• song-wx / SIFT
  [ICML2024 Spotlight] Fine-Tuning Pre-trained Large Language Models Sparsely
  ☆13Updated 2 months ago
• NUS-HPC-AI-Lab / LARS-ImageNet-PyTorch
  Accuracy 77%. Large batch deep learning optimizer LARS for ImageNet with PyTorch and ResNet, using Horovod for distribution. Optional acc…
  ☆37Updated 3 years ago
• varunnair18 / FISH
  Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).
  ☆54Updated 2 years ago
• HKUNLP / efficient-attention
  [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling
  ☆78Updated last year
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆42Updated last year
• dguo98 / DiffPruning
  Parameter Efficient Transfer Learning with Diff Pruning
  ☆70Updated 3 years ago
• microsoft / ReinMax
  Beyond Straight-Through
  ☆82Updated last year
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆24Updated 5 months ago
• NUS-HPC-AI-Lab / pytorch-lamb
  PyTorch implementation of LAMB for ImageNet/ResNet-50 training
  ☆13Updated 3 years ago
• alexzhang13 / flashattention2-custom-mask
  Triton implementation of FlashAttention2 that adds Custom Masks.
  ☆62Updated last month
• microsoft / Stochastic-Mixture-of-Experts
  This package implements THOR: Transformer with Stochastic Experts.
  ☆60Updated 2 years ago
• nik-dim / tall_masks
  Official repository of "Localizing Task Information for Improved Model Merging and Compression" [ICML 2024]
  ☆27Updated 4 months ago
• QingruZhang / PLATON
  This pytorch package implements PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance (ICML 2022).
  ☆39Updated last year
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆104Updated 6 months ago
• hwang595 / Cuttlefish
  The implementation for MLSys 2023 paper: "Cuttlefish: Low-rank Model Training without All The Tuning"
  ☆42Updated last year