Alternatives and similar repositories for Transformer-Workbench:

Users that are interested in Transformer-Workbench are comparing it to the libraries listed below

• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆50Updated 6 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 6 months ago
• nyunAI / Faster-LLM-Survey
  ☆41Updated last year
• YeonwooSung / Pytorch_mixture-of-experts
  PyTorch implementation of moe, which stands for mixture of experts
  ☆43Updated 4 years ago
• kyegomez / MultiQueryAttention
  This is a simple torch implementation of the high performance Multi-Query Attention
  ☆16Updated last year
• Nicolas-BZRD / llm-recipes
  ☆28Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆90Updated this week
• tanaymeh / mamba-train
  A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM
  ☆54Updated last year
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated last year
• GeneZC / MiniMA
  Code for paper titled "Towards the Law of Capacity Gap in Distilling Language Models"
  ☆100Updated 9 months ago
• daniel-furman / sft-demos
  Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
  ☆74Updated 6 months ago
• IST-DASLab / RoSA
  Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)
  ☆40Updated last year
• Percent-BFD / neurips_submission
  ☆15Updated last year
• alperiox / Compact-Language-Models-via-Pruning-and-Knowledge-Distillation
  Unofficial implementation of https://arxiv.org/pdf/2407.14679
  ☆44Updated 7 months ago
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆52Updated 3 weeks ago
• Upaya07 / NeurIPS-llm-efficiency-challenge
  Code for NeurIPS LLM Efficiency Challenge
  ☆57Updated last year
• thepowerfuldeez / OLMo
  My fork os allen AI's OLMo for educational purposes.
  ☆30Updated 4 months ago
• princeton-pli / MeCo
  Code for preprint "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"
  ☆36Updated last month
• ElleLeonne / Lightning-ReLoRA
  A public implementation of the ReLoRA pretraining method, built on Lightning-AI's Pytorch Lightning suite.
  ☆33Updated last year
• aju22 / LLaMA2
  This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…
  ☆64Updated last year
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆63Updated last year
• Pleias / Various-Finetuning
  Set of scripts to finetune LLMs
  ☆37Updated last year
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆135Updated 9 months ago
• tridao / flash-attention-wheels
  ☆48Updated last year
• thu-coai / MiniPLM
  [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Models
  ☆40Updated 5 months ago
• NielsRogge / awesome-huggingface
  Repository containing awesome resources regarding Hugging Face tooling.
  ☆46Updated last year
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 3 months ago
• kyegomez / Infini-attention
  Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…
  ☆55Updated this week
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 7 months ago
• cooper12121 / llama3-8x8b-MoE
  Copy the MLP of llama3 8 times as 8 experts , created a router with random initialization,add load balancing loss to construct an 8x8b Mo…
  ☆26Updated 9 months ago