Alternatives and similar repositories for RethinkTinyLM:

Users that are interested in RethinkTinyLM are comparing it to the libraries listed below

• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆99Updated 7 months ago
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆145Updated 7 months ago
• SparkJiao / llama-pipeline-parallel
  A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…
  ☆53Updated last year
• 18907305772 / FuseAI
  FuseAI Project
  ☆80Updated this week
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆128Updated 7 months ago
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆76Updated 10 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆125Updated last year
• TemporaryLoRA / Temp-LoRA
  ☆95Updated 9 months ago
• SkyworkAI / skywork-o1-prm-inference
  ☆55Updated 2 months ago
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆70Updated last year
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆96Updated 4 months ago
• Glaciohound / LM-Infinite
  Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"
  ☆134Updated 3 months ago
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆77Updated 2 months ago
• yegcjs / mixinglaws
  ☆94Updated 4 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated 9 months ago
• jongwooko / distillm
  Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)
  ☆173Updated 4 months ago
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆154Updated 7 months ago
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆62Updated last year
• microsoft / LongRoPE
  LongRoPE is a novel method that can extends the context window of pre-trained LLMs to an impressive 2048k tokens.
  ☆123Updated 5 months ago
• ScalingIntelligence / large_language_monkeys
  ☆80Updated 4 months ago
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆149Updated last month
• fxmeng / mixtral_spliter
  Converting Mixtral-8x7B to Mixtral-[1~7]x7B
  ☆20Updated 10 months ago
• hkust-nlp / llm-compression-intelligence
  Official github repo for the paper "Compression Represents Intelligence Linearly" [COLM 2024]
  ☆130Updated 4 months ago
• pan-x-c / EE-LLM
  EE-LLM is a framework for large-scale training and inference of early-exit (EE) large language models (LLMs).
  ☆52Updated 7 months ago
• thu-nics / MoA
  The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>
  ☆113Updated last month
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆125Updated 6 months ago
• SalesforceAIResearch / GemFilter
  ☆72Updated 2 weeks ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆84Updated 3 months ago
• xijiu9 / Train_Transformers_with_INT4
  ☆140Updated last year
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆76Updated 7 months ago