Alternatives and similar repositories for RethinkTinyLM

Users that are interested in RethinkTinyLM are comparing it to the libraries listed below

• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆78Updated last year
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆101Updated last year
• SparkJiao / llama-pipeline-parallel
  A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…
  ☆55Updated last year
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆154Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆133Updated last year
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆85Updated 7 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• pprp / Awesome-Efficient-MoE
  Efficient Mixture of Experts for LLM Paper List
  ☆72Updated 6 months ago
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆64Updated last year
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆158Updated 2 months ago
• yegcjs / mixinglaws
  ☆101Updated 8 months ago
• TemporaryLoRA / Temp-LoRA
  ☆105Updated last year
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆163Updated last year
• Glaciohound / LM-Infinite
  Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"
  ☆146Updated 3 months ago
• 18907305772 / FuseAI
  FuseAI Project
  ☆87Updated 4 months ago
• timinar / BabyLlama
  Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.
  ☆80Updated last year
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆73Updated last year
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆143Updated 9 months ago
• QwenLM / online_merging_optimizers
  Implementations of online merging optimizers proposed by Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment
  ☆75Updated last year
• infinigence / LVEval
  Repository of LV-Eval Benchmark
  ☆67Updated 9 months ago
• sramshetty / ShortGPT
  Unofficial implementations of block/layer-wise pruning methods for LLMs.
  ☆70Updated last year
• SkyworkAI / skywork-o1-prm-inference
  ☆63Updated 6 months ago
• thu-coai / MiniPLM
  [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Models
  ☆47Updated 7 months ago
• OpenSparseLLMs / MoM
  ☆84Updated 2 months ago
• OpenSparseLLMs / Linear-MoE
  ☆104Updated 2 weeks ago
• KaihuaTang / Qwen-Tokenizer-Pruner
  Due to the huge vocaburary size (151,936) of Qwen models, the Embedding and LM Head weights are excessively heavy. Therefore, this projec…
  ☆22Updated 10 months ago
• berlino / gated_linear_attention
  ☆104Updated last year
• Zanette-Labs / SpeculativeRejection
  [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejection
  ☆45Updated 7 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated last year
• RUCAIBox / QuantizedEmpirical
  ☆14Updated last year