Alternatives and similar repositories for Tokenizer-Changer

Users that are interested in Tokenizer-Changer are comparing it to the libraries listed below

• erogol / BlaGPT
  Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible…
  ☆89Updated last month
• ictnlp / SiLLM
  SiLLM is a Simultaneous Machine Translation (SiMT) Framework. It utilizes a Large Language model as the translation model and employs a t…
  ☆17Updated last year
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆95Updated last month
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆142Updated 11 months ago
• WowCZ / LongMIT
  LongMIT: Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets
  ☆40Updated last year
• chtmp223 / suri
  Suri: Multi-constraint instruction following for long-form text generation (EMNLP’24)
  ☆27Updated 2 months ago
• princeton-pli / MeCo
  Code for ICML 25 paper "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"
  ☆48Updated 5 months ago
• teelinsan / parallel-decoding
  Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"
  ☆121Updated last year
• gmftbyGMFTBY / Rep-Dropout
  [NeurIPS 2023] Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
  ☆38Updated 2 years ago
• Leooyii / LCEG
  Long Context Extension and Generalization in LLMs
  ☆62Updated last year
• hamishivi / tess-2
  Repository for "TESS-2: A Large-Scale, Generalist Diffusion Language Model"
  ☆53Updated 9 months ago
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆78Updated last year
• NJUNLP / MMT-LLM
  ☆35Updated 2 years ago
• SalesforceAIResearch / GemFilter
  ☆85Updated last month
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆126Updated 11 months ago
• danny911kr / REALTALK
  Evaluate your agent memory on real-world dialogues, not LLM-simulated dialogues.
  ☆35Updated 5 months ago
• DRSY / KV_Compression
  [EMNLP 2023]Context Compression for Auto-regressive Transformers with Sentinel Tokens
  ☆25Updated 2 years ago
• wxjiao / InstructMT
  A collection of instruction data and scripts for machine translation.
  ☆20Updated 2 years ago
• PythonNut / superbpe
  Official code release for "SuperBPE: Space Travel for Language Models"
  ☆76Updated 3 weeks ago
• john-hewitt / backpacks-flash-attn
  The original Backpack Language Model implementation, a fork of FlashAttention
  ☆69Updated 2 years ago
• jasonyux / TriPosT
  ☆12Updated last year
• zhuzilin / vllm-group
  ☆12Updated last year
• chujiezheng / LLM-Extrapolation
  Official repository for ACL 2025 paper "Model Extrapolation Expedites Alignment"
  ☆76Updated 6 months ago
• lukemelas / mtob
  ☆43Updated last year
• hlt-mt / FBK-fairseq
  Repository containing the open source code of works published at the FBK MT unit.
  ☆56Updated last month
• chijames / KERPLE
  ☆20Updated 3 years ago
• CONE-MT / LLaMAX
  ☆72Updated last year
• epfl-dlab / llm-latent-language
  Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".
  ☆80Updated last year
• raymin0223 / fast_robust_early_exit
  Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)
  ☆64Updated last year
• VITA-Group / Ms-PoE
  "Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiw…
  ☆30Updated last year