Alternatives and similar repositories for bytepiece-rs

Users that are interested in bytepiece-rs are comparing it to the libraries listed below

• RWKV-Wiki / MultilingualShareGPT
  MultilingualShareGPT, the free multi-language corpus for LLM training
  ☆73Updated 2 years ago
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆58Updated last year
• Jellyfish042 / uncheatable_eval
  Evaluating LLMs with Dynamic Data
  ☆99Updated 4 months ago
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆64Updated 2 years ago
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆73Updated 2 years ago
• codefuse-ai / Collinear-Constrained-Attention
  ☆62Updated last year
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆33Updated last year
• TemporaryLoRA / Temp-LoRA
  ☆122Updated last year
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆41Updated last year
• princeton-pli / MeCo
  Code for ICML 25 paper "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"
  ☆48Updated 5 months ago
• juvi21 / CoPE-cuda
  Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719
  ☆22Updated last year
• OpenMOSE / RWKV-Infer
  A large-scale RWKV v7(World, PRWKV, Hybrid-RWKV) inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to deploy…
  ☆45Updated last month
• cofe-ai / Mu-scaling
  Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales
  ☆32Updated 2 years ago
• AI21Labs / Parallel-Context-Windows
  ☆105Updated 2 years ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆137Updated last year
• ExpressAI / AI-Gaokao
  Gaokao Benchmark for AI
  ☆109Updated 3 years ago
• Oneflow-Inc / one-glm
  A more efficient GLM implementation!
  ☆54Updated 2 years ago
• OpenLMLab / MOSS_WebSearchTool
  MOSS 003 WebSearchTool: A simple but reliable implementation
  ☆45Updated 2 years ago
• THUDM / ChatGLM-Math
  ☆83Updated last year
• Academic-Hammer / HammerLLM
  1.4B sLLM for Chinese and English - HammerLLM🔨
  ☆43Updated last year
• Bui1dMySea / MemLong
  ☆95Updated last year
• woct0rdho / transformers-qwen3-moe-fused
  Fused Qwen3 MoE layer for faster training, compatible with HF Transformers, LoRA, 4-bit quant, Unsloth
  ☆212Updated last month
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆249Updated last year
• jungokasai / T2R
  ☆14Updated 3 years ago
• FlagOpen / Infinity-Instruct
  ☆50Updated last year
• THUDM / icetk
  A unified tokenization tool for Images, Chinese and English.
  ☆153Updated 2 years ago
• Jacob-Zhou / simple-csc
  This repository provides an implementation of "A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction B…
  ☆84Updated 4 months ago
• microsoft / nanovppo
  Nano repo for RL training of LLMs
  ☆69Updated last month
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆97Updated last year
• kyegomez / Infini-attention
  Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…
  ☆58Updated this week