Alternatives and similar repositories for bytepiece-rs

Users that are interested in bytepiece-rs are comparing it to the libraries listed below

• RWKV-Wiki / MultilingualShareGPT
  MultilingualShareGPT, the free multi-language corpus for LLM training
  ☆72Updated 2 years ago
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆37Updated last year
• Jellyfish042 / uncheatable_eval
  Evaluating LLMs with Dynamic Data
  ☆92Updated 2 weeks ago
• hscspring / bytepiece-rs
  The Bytepiece Tokenizer Implemented in Rust.
  ☆14Updated last year
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆132Updated 11 months ago
• TemporaryLoRA / Temp-LoRA
  ☆105Updated last year
• FlagOpen / Infinity-Instruct
  ☆47Updated 11 months ago
• microsoft / nanovppo
  Nano repo for RL training of LLMs
  ☆60Updated last week
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆73Updated last year
• princeton-pli / MeCo
  Code for preprint "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"
  ☆39Updated last month
• beichao1314 / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆65Updated 2 years ago
• Academic-Hammer / HammerLLM
  1.4B sLLM for Chinese and English - HammerLLM🔨
  ☆44Updated last year
• ExpressAI / AI-Gaokao
  Gaokao Benchmark for AI
  ☆108Updated 2 years ago
• THUDM / icetk
  A unified tokenization tool for Images, Chinese and English.
  ☆152Updated 2 years ago
• kyegomez / Infini-attention
  Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…
  ☆55Updated this week
• SomeoneKong / llm_long_context_bench202405
  ☆29Updated 9 months ago
• Oxer11 / Crosstalk-Generation
  Perform crosstalk with Qian Yu
  ☆54Updated last year
• cofe-ai / Mu-scaling
  Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales
  ☆32Updated last year
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆32Updated last year
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆53Updated last year
• aplmikex / deduplication_mnbvc
  文本去重
  ☆72Updated last year
• wangrui6 / Zhihu-KOL
  ☆31Updated 2 years ago
• sheryc / resonance_rope
  [ACL 24 Findings] Implementation of Resonance RoPE and the PosGen synthetic dataset.
  ☆22Updated last year
• airaria / GRAIN
  GRAIN: Gradient-based Intra-attention Pruning on Pre-trained Language Models
  ☆19Updated last year
• SUSTech-IDEA / SUS-Chat
  SUS-Chat: Instruction tuning done right
  ☆48Updated last year
• zexuanqiu / CLongEval
  CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
  ☆40Updated last year
• Jacob-Zhou / simple-csc
  This repository provides an implementation of the paper "A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Co…
  ☆70Updated 2 months ago
• Zheng0428 / COIG-Kun
  ☆36Updated 9 months ago
• bojone / tiger
  A Tight-fisted Optimizer
  ☆48Updated 2 years ago