IBM / ModuleFormer
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
☆217Updated 11 months ago
Alternatives and similar repositories for ModuleFormer:
Users that are interested in ModuleFormer are comparing it to the libraries listed below
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆205Updated 10 months ago
- TART: A plug-and-play Transformer module for task-agnostic reasoning☆195Updated last year
- Code repository for the c-BTM paper☆106Updated last year
- Multipack distributed sampler for fast padding-free training of LLMs☆186Updated 7 months ago
- Experiments on speculative sampling with Llama models☆125Updated last year
- ☆94Updated last year
- Evaluating LLMs with CommonGen-Lite☆89Updated last year
- Simple next-token-prediction for RLHF☆223Updated last year
- The official repo for "LLoCo: Learning Long Contexts Offline"☆116Updated 9 months ago
- Manage scalable open LLM inference endpoints in Slurm clusters☆253Updated 8 months ago
- Spherical Merge Pytorch/HF format Language Models with minimal feature loss.☆119Updated last year
- batched loras☆340Updated last year
- This is the repo for the paper Shepherd -- A Critic for Language Model Generation☆218Updated last year
- Official repository for "Scaling Retrieval-Based Langauge Models with a Trillion-Token Datastore".☆195Updated this week
- ☆412Updated last year
- ☆125Updated last year
- Mixing Language Models with Self-Verification and Meta-Verification☆102Updated 3 months ago
- Learning to Compress Prompts with Gist Tokens - https://arxiv.org/abs/2304.08467☆283Updated last month
- Scaling Data-Constrained Language Models☆335Updated 6 months ago
- ☆131Updated last year
- EvaByte: Efficient Byte-level Language Models at Scale☆85Updated 2 weeks ago
- ☆119Updated 6 months ago
- Self-Alignment with Principle-Following Reward Models☆156Updated last year
- Code for NeurIPS'24 paper 'Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization'☆186Updated 4 months ago
- ☆150Updated last year
- Flacuna was developed by fine-tuning Vicuna on Flan-mini, a comprehensive instruction collection encompassing various tasks. Vicuna is al…☆111Updated last year
- Code and data accompanying our paper on arXiv "Faithful Chain-of-Thought Reasoning".☆157Updated 10 months ago
- Code for PHATGOOSE introduced in "Learning to Route Among Specialized Experts for Zero-Shot Generalization"☆82Updated last year
- DSIR large-scale data selection framework for language model training☆244Updated 11 months ago
- Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".☆272Updated last year