Alternatives and similar repositories for Transformer-Workbench:

Users that are interested in Transformer-Workbench are comparing it to the libraries listed below

• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆47Updated 5 months ago
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆159Updated 10 months ago
• thepowerfuldeez / OLMo
  My fork os allen AI's OLMo for educational purposes.
  ☆30Updated 3 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆71Updated last year
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 7 months ago
• juvi21 / CoPE-cuda
  Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719
  ☆22Updated 9 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 6 months ago
• alperiox / Compact-Language-Models-via-Pruning-and-Knowledge-Distillation
  Unofficial implementation of https://arxiv.org/pdf/2407.14679
  ☆44Updated 6 months ago
• rasbt / cvpr2023
  ☆131Updated last year
• YeonwooSung / Pytorch_mixture-of-experts
  PyTorch implementation of moe, which stands for mixture of experts
  ☆42Updated 4 years ago
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆150Updated 3 months ago
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated 11 months ago
• kjslag / spacebyte
  A byte-level decoder architecture that matches the performance of tokenized Transformers.
  ☆63Updated 11 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆86Updated 2 weeks ago
• The-AI-Summer / pytorch-ddp
  code for the ddp tutorial
  ☆32Updated 2 years ago
• Nicolas-BZRD / llm-recipes
  ☆29Updated last year
• kyegomez / MHMoE
  Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch
  ☆22Updated 2 months ago
• kyegomez / TTL
  Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"
  ☆24Updated last week
• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆80Updated 7 months ago
• RamonKaspar / MathPrompter
  MathPrompter Implementation: This repository hosts an implementation based on the 'MathPrompter: Mathematical Reasoning Using Large Langu…
  ☆13Updated 8 months ago
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆52Updated 2 months ago
• rasbt / dora-from-scratch
  LoRA and DoRA from Scratch Implementations
  ☆199Updated last year
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆87Updated last year
• Arenaa / Accelerated-Generation-Techniques
  This repository contains papers for a comprehensive survey on accelerated generation techniques in Large Language Models (LLMs).
  ☆11Updated 10 months ago
• NX-AI / mlstm_kernels
  Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.
  ☆47Updated last week
• mistralai / mistral-evals
  ☆66Updated last week
• nengelmann / Fuyu-8B---Exploration
  Exploration of the multi modal fuyu-8b model of Adept. 🤓 🔍
  ☆28Updated last year
• tanaymeh / mamba-train
  A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM
  ☆54Updated 11 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 2 months ago
• haochengxi / Train_Transformers_with_INT4
  ☆145Updated last year