Alternatives and similar repositories for Megatron-DeepSpeed

Users that are interested in Megatron-DeepSpeed are comparing it to the libraries listed below

• shayne-longpre / a-pretrainers-guide
  ☆72Updated 2 years ago
• seonghyeonye / Flipped-Learning
  [ICLR 2023] Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners
  ☆116Updated 7 months ago
• allenai / bff
  ☆38Updated last year
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆142Updated last year
• huggingface / olm-training
  Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.
  ☆96Updated 3 years ago
• huggingface / that_is_good_data
  ☆65Updated 2 years ago
• mega002 / lm-debugger
  The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.
  ☆182Updated 3 years ago
• huggingface / olm-datasets
  Pipeline for pulling and processing online language model pretraining data from the web
  ☆177Updated 2 years ago
• google-research / t5x_retrieval
  ☆102Updated 3 years ago
• Rallio67 / language-model-agents
  Experiments with generating opensource language model assistants
  ☆97Updated 2 years ago
• huggingface / transformers_bloom_parallel
  Techniques used to run BLOOM at inference in parallel
  ☆37Updated 3 years ago
• frankxu2004 / knnlm-why
  Repo for ICML23 "Why do Nearest Neighbor Language Models Work?"
  ☆59Updated 3 years ago
• kernelmachine / silo-lm
  SILO Language Models code repository
  ☆83Updated last year
• EleutherAI / semantic-memorization
  ☆44Updated last year
• zphang / minimal-opt
  ☆67Updated 3 years ago
• ZurichNLP / mbr
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆60Updated last year
• bigscience-workshop / metadata
  Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.
  ☆31Updated 2 years ago
• bigscience-workshop / lm-evaluation-harness
  A framework for few-shot evaluation of autoregressive language models.
  ☆106Updated 2 years ago
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆87Updated 2 weeks ago
• microsoft / mutransformers
  some common Huggingface transformers in maximal update parametrization (µP)
  ☆87Updated 3 years ago
• hadasah / btm
  ☆77Updated last year
• google-research / xtreme-up
  ☆52Updated 2 years ago
• thakur-nandan / income
  INCOME: An Easy Repository for Training and Evaluation of Index Compression Methods in Dense Retrieval. Includes BPR and JPQ.
  ☆24Updated 2 years ago
• konstantinjdobler / focus
  [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"
  ☆36Updated 8 months ago
• zorazrw / odex
  [EMNLP'23] Execution-Based Evaluation for Open Domain Code Generation
  ☆49Updated 2 years ago
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆49Updated 2 years ago
• facebookresearch / NPM
  The original implementation of Min et al. "Nonparametric Masked Language Modeling" (paper https//arxiv.org/abs/2212.01349)
  ☆158Updated 3 years ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆27Updated last year
• marzenakrp / nocha
  ☆54Updated last year
• kernelmachine / cbtm
  Code repository for the c-BTM paper
  ☆108Updated 2 years ago