bigscience-workshop / Megatron-DeepSpeed

Related projects ⓘ

Alternatives and complementary repositories for Megatron-DeepSpeed

• microsoft / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,893Updated last month
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆560Updated last month
• bigscience-workshop / bigscience
  Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.
  ☆980Updated 3 months ago
• alibaba / Megatron-LLaMA
  Best practice for training LLaMA models in Megatron-LM
  ☆628Updated 10 months ago
• IST-DASLab / gptq
  Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".
  ☆1,941Updated 7 months ago
• microsoft / Tutel
  Tutel MoE: An Optimized Mixture-of-Experts Implementation
  ☆735Updated this week
• microsoft / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆1,904Updated this week
• AetherCortex / Llama-X
  Open Academic Research on Improving LLaMA to SOTA LLM
  ☆1,607Updated last year
• hao-ai-lab / LookaheadDecoding
  [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  ☆1,149Updated last month
• GanjinZero / RRHF
  [NIPS2023] RRHF & Wombat
  ☆798Updated last year
• bigscience-workshop / xmtf
  Crosslingual Generalization through Multitask Finetuning
  ☆516Updated last month
• google-research / FLAN
  ☆1,474Updated 3 weeks ago
• hendrycks / test
  Measuring Massive Multitask Language Understanding | ICLR 2021
  ☆1,216Updated last year
• epfLLM / Megatron-LLM
  distributed trainer for LLMs
  ☆545Updated 6 months ago
• OpenBMB / BMTrain
  Efficient Training (including pre-training and fine-tuning) for Big Models
  ☆564Updated 3 months ago
• OpenLMLab / LOMO
  LOMO: LOw-Memory Optimization
  ☆979Updated 4 months ago
• allenai / natural-instructions
  Expanding natural instructions
  ☆959Updated 11 months ago
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆306Updated last year
• NVIDIA / TransformerEngine
  A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs…
  ☆1,979Updated this week
• jquesnelle / yarn
  YaRN: Efficient Context Window Extension of Large Language Models
  ☆1,353Updated 7 months ago
• BlackSamorez / tensor_parallel
  Automatically split your PyTorch models on multiple GPUs for training & inference
  ☆626Updated 10 months ago
• anthropics / hh-rlhf
  Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"
  ☆1,624Updated last year
• ofirpress / attention_with_linear_biases
  Code for the ALiBi method for transformer language models (ICLR 2022)
  ☆507Updated last year
• stanford-crfm / helm
  Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09…
  ☆1,948Updated this week
• alibaba / Pai-Megatron-Patch
  The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.
  ☆721Updated this week
• allenai / RL4LMs
  A modular RL library to fine-tune language models to human preferences
  ☆2,213Updated 8 months ago
• deepseek-ai / DeepSeek-MoE
  DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
  ☆1,008Updated 10 months ago
• AGI-Edgerunners / LLM-Adapters
  Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"
  ☆1,079Updated 8 months ago
• THUDM / P-tuning
  A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.
  ☆923Updated 2 years ago