IBM / ModuleFormer
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
☆220Updated last year
Alternatives and similar repositories for ModuleFormer:
Users that are interested in ModuleFormer are comparing it to the libraries listed below
- The official repo for "LLoCo: Learning Long Contexts Offline"☆116Updated 10 months ago
- Experiments on speculative sampling with Llama models☆125Updated last year
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆205Updated 11 months ago
- Code repository for the c-BTM paper☆106Updated last year
- Multipack distributed sampler for fast padding-free training of LLMs☆188Updated 8 months ago
- Evaluating LLMs with CommonGen-Lite☆89Updated last year
- ☆94Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆254Updated 9 months ago
- TART: A plug-and-play Transformer module for task-agnostic reasoning☆196Updated last year
- Simple next-token-prediction for RLHF☆225Updated last year
- ☆255Updated last year
- Spherical Merge Pytorch/HF format Language Models with minimal feature loss.☆120Updated last year
- Code and Data for "Long-context LLMs Struggle with Long In-context Learning" [TMLR2025]☆105Updated 2 months ago
- Pre-training code for Amber 7B LLM☆166Updated 11 months ago
- Scripts for generating synthetic finetuning data for reducing sycophancy.☆110Updated last year
- This is the repo for the paper Shepherd -- A Critic for Language Model Generation☆218Updated last year
- DSIR large-scale data selection framework for language model training☆246Updated last year
- [EMNLP 2023] The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning☆238Updated last year
- ☆132Updated last year
- LongEmbed: Extending Embedding Models for Long Context Retrieval (EMNLP 2024)☆133Updated 5 months ago
- Just a bunch of benchmark logs for different LLMs☆119Updated 8 months ago
- Data preparation code for Amber 7B LLM☆88Updated 11 months ago
- ☆197Updated 4 months ago
- Official repository for "Scaling Retrieval-Based Langauge Models with a Trillion-Token Datastore".☆196Updated 2 weeks ago
- Self-Alignment with Principle-Following Reward Models☆160Updated last year
- ☆412Updated last year
- ☆126Updated 5 months ago
- ☆125Updated last year
- Benchmarking LLMs with Challenging Tasks from Real Users☆221Updated 5 months ago
- Mixing Language Models with Self-Verification and Meta-Verification☆104Updated 4 months ago