Montinger / Transformer-WorkbenchLinks
Playground for Transformers
☆51Updated last year
Alternatives and similar repositories for Transformer-Workbench
Users that are interested in Transformer-Workbench are comparing it to the libraries listed below
Sorting:
- several types of attention modules written in PyTorch for learning purposes☆53Updated 10 months ago
- PyTorch implementation of moe, which stands for mixture of experts☆46Updated 4 years ago
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆98Updated 10 months ago
- Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M cont…☆83Updated last year
- code for the ddp tutorial☆32Updated 3 years ago
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆58Updated last year
- ☆42Updated last year
- LoRA and DoRA from Scratch Implementations☆208Updated last year
- minimal GRPO implementation from scratch☆94Updated 4 months ago
- Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.☆78Updated 9 months ago
- minimal scripts for 24GB VRAM GPUs. training, inference, whatever☆41Updated last month
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆123Updated 6 months ago
- Scripts for text classification with llama and bert☆22Updated 2 weeks ago
- Root Mean Square Layer Normalization☆249Updated 2 years ago
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆76Updated last year
- ☆20Updated 4 years ago
- Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind☆177Updated 10 months ago
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆120Updated 9 months ago
- 📚 Text Classification with LoRA (Low-Rank Adaptation) of Language Models - Efficiently fine-tune large language models for text classifi…☆52Updated last year
- Implementation of the Mamba SSM with hf_integration.☆56Updated 11 months ago
- A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.☆94Updated 7 months ago
- Fast instruction tuning with Llama2☆11Updated last year
- First-principle implementations of groundbreaking AI algorithms using a wide range of deep learning frameworks, accompanied by supporting…☆178Updated 3 weeks ago
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper☆33Updated last year
- (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …☆175Updated last year
- Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.☆81Updated last year
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆160Updated 3 months ago
- a curated list of the role of small models in the LLM era☆103Updated 10 months ago
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆70Updated last year
- ☆48Updated 11 months ago