Alternatives and similar repositories for LLaMA-Megatron

Users that are interested in LLaMA-Megatron are comparing it to the libraries listed below

• CoinCheung / gdGPT
  Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
  ☆95Updated last year
• ProjectD-AI / LLaMA-Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆69Updated last year
• genggui001 / Megatron-DeepSpeed-Llama
  ☆84Updated last year
• zexuanqiu / CLongEval
  CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
  ☆40Updated last year
• keezen / ntk_alibi
  NTK scaled version of ALiBi position encoding in Transformer.
  ☆68Updated last year
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆73Updated last year
• infinigence / LVEval
  Repository of LV-Eval Benchmark
  ☆65Updated 8 months ago
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆62Updated last year
• seanzhang-zhichen / baichuan-Dynamic-NTK-ALiBi
  百川Dynamic NTK-ALiBi的代码实现：无需微调即可推理更长文本
  ☆47Updated last year
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆153Updated 11 months ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆103Updated last month
• SparkJiao / llama-pipeline-parallel
  A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…
  ☆55Updated last year
• SkyworkAI / skywork-o1-prm-inference
  ☆63Updated 5 months ago
• l294265421 / alpaca-rlhf
  Finetuning LLaMA with RLHF (Reinforcement Learning with Human Feedback) based on DeepSpeed Chat
  ☆115Updated last year
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆324Updated 7 months ago
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆64Updated last year
• alibaba / ChatLearn
  A flexible and efficient training framework for large-scale alignment tasks
  ☆346Updated 3 months ago
• IronBeliever / CaR
  Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation
  ☆77Updated 6 months ago
• yegcjs / mixinglaws
  ☆98Updated 7 months ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆219Updated last year
• RUC-GSAI / Llama-3-SynE
  Llama-3-SynE: A Significantly Enhanced Version of Llama-3 with Advanced Scientific Reasoning and Chinese Language Capabilities | 继续预训练提升 …
  ☆32Updated 5 months ago
• RUCAIBox / EASYEP
  ☆18Updated last month
• madsys-dev / deepseekv2-profile
  ☆132Updated 2 months ago
• FlagOpen / Infinity-Instruct
  ☆46Updated 11 months ago
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆209Updated 8 months ago
• THUDM / LongAlign
  [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs
  ☆249Updated 5 months ago
• MayDomine / Burst-Attention
  Distributed IO-aware Attention algorithm
  ☆20Updated 8 months ago
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated last year
• anyscale / llm-continuous-batching-benchmarks
  ☆117Updated last year
• tianyi-lab / Superfiltering
  [ACL'24] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
  ☆151Updated 8 months ago