Alternatives and similar repositories for MatFormer-OLMo:

Users that are interested in MatFormer-OLMo are comparing it to the libraries listed below

• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆43Updated 7 months ago
• epfml / pam
  ☆15Updated last year
• pixeli99 / MixLN
  [ICLR 2025] Official Pytorch Implementation of "Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN" by Pengxia…
  ☆21Updated 4 months ago
• shoaibahmed / llm_depth_pruning
  Official implementation of the paper: "A deeper look at depth pruning of LLMs"
  ☆15Updated 9 months ago
• IST-DASLab / RoSA
  Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)
  ☆41Updated last year
• prateeky2806 / ComPEFT
  ☆25Updated last year
• thunlp / SparsingLaw
  The open-source materials for paper "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity".
  ☆21Updated 5 months ago
• IST-DASLab / MicroAdam
  This repository contains code for the MicroAdam paper.
  ☆18Updated 4 months ago
• dkopi / Bitune
  Implementation of Bitune: Bidirectional Instruction-Tuning
  ☆19Updated 11 months ago
• nverma1 / merging-text-transformers
  Code for "Merging Text Transformers from Different Initializations"
  ☆20Updated 3 months ago
• facebookresearch / adaptive_scheduling
  Experimental scripts for researching data adaptive learning rate scheduling.
  ☆23Updated last year
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆40Updated last year
• smonsays / hypernetwork-attention
  Official code for the paper "Attention as a Hypernetwork"
  ☆32Updated 10 months ago
• GATECH-EIC / Linearized-LLM
  [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  ☆30Updated 10 months ago
• bentherien / mu_learned_optimization
  [Oral; Neurips OPT2024 ] μLO: Compute-Efficient Meta-Generalization of Learned Optimizers
  ☆12Updated last month
• VITA-Group / LoCoCo
  [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen
  ☆16Updated 8 months ago
• stan-anony / derivative_free_lora_rank
  ☆16Updated last year
• Qichuzyy / POA
  Official implementation of ECCV24 paper: POA
  ☆24Updated 9 months ago
• PiotrNawrot / sparse-frontier
  Official implementation of "The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs"
  ☆26Updated 2 weeks ago
• tianyi-lab / C3PO
  Code for "C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing"
  ☆14Updated last month
• locuslab / T-MARS
  Code for T-MARS data filtering
  ☆35Updated last year
• srush / LLM-Talk
  ☆49Updated last year
• mobiusml / low-rank-llama2
  Low-Rank Llama Custom Training
  ☆22Updated last year
• kyleliang919 / Online-Subspace-Descent
  This repo is based on https://github.com/jiaweizzhao/GaLore
  ☆27Updated 7 months ago
• renll / SeqBoat
  [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling
  ☆36Updated last year
• facebookresearch / Ternary_Binary_Transformer
  ACL 2023
  ☆39Updated last year
• tianyi-lab / R2-T2
  Code for "R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts"
  ☆15Updated 2 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆27Updated last year
• HazyResearch / aioli
  Aioli: A unified optimization framework for language model data mixing
  ☆25Updated 3 months ago
• sail-sg / SkyLadder
  The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling
  ☆29Updated last month