Alternatives and similar repositories for Tokenizer-Changer

Users that are interested in Tokenizer-Changer are comparing it to the libraries listed below

• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆142Updated 11 months ago
• chtmp223 / suri
  Suri: Multi-constraint instruction following for long-form text generation (EMNLP’24)
  ☆27Updated 3 months ago
• teelinsan / parallel-decoding
  Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"
  ☆121Updated last year
• WowCZ / LongMIT
  LongMIT: Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets
  ☆39Updated last year
• erogol / BlaGPT
  Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible…
  ☆92Updated 3 weeks ago
• princeton-pli / MeCo
  Code for ICML 25 paper "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"
  ☆48Updated 6 months ago
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆96Updated last month
• gmftbyGMFTBY / Rep-Dropout
  [NeurIPS 2023] Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
  ☆39Updated 2 years ago
• qiuzh20 / EMoE
  Official PyTorch Implementation of EMoE: Unlocking Emergent Modularity in Large Language Models [main conference @ NAACL2024]
  ☆38Updated last year
• chujiezheng / LLM-Extrapolation
  Official repository for ACL 2025 paper "Model Extrapolation Expedites Alignment"
  ☆76Updated 7 months ago
• jasonyux / TriPosT
  ☆12Updated last year
• Leooyii / LCEG
  Long Context Extension and Generalization in LLMs
  ☆62Updated last year
• ictnlp / SiLLM
  SiLLM is a Simultaneous Machine Translation (SiMT) Framework. It utilizes a Large Language model as the translation model and employs a t…
  ☆17Updated last year
• wxjiao / InstructMT
  A collection of instruction data and scripts for machine translation.
  ☆20Updated 2 years ago
• NJUNLP / MMT-LLM
  ☆35Updated 2 years ago
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆78Updated last year
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆33Updated last year
• CONE-MT / LLaMAX
  ☆72Updated last year
• facebookresearch / worldsense
  WorldSense benchmark for grounded reasoning in language models
  ☆22Updated 2 years ago
• lukemelas / mtob
  ☆44Updated last year
• rbawden / mt-bigscience
  Evaluation results for Machine Translation within the BigScience project
  ☆11Updated 2 years ago
• dadelani / sib-200
  SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
  ☆23Updated 11 months ago
• cisnlp / Glot500
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆107Updated last year
• TrustJudge / TrustJudge
  TrustJudge is a probabilistic evaluation framework that reduces score-comparison and pairwise transitivity inconsistencies in LLM-as-a-ju…
  ☆38Updated 3 months ago
• SalesforceAIResearch / GemFilter
  ☆85Updated last month
• microsoft / nanovppo
  Nano repo for RL training of LLMs
  ☆70Updated 2 months ago
• danny911kr / REALTALK
  Evaluate your agent memory on real-world dialogues, not LLM-simulated dialogues.
  ☆35Updated 6 months ago
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆29Updated last year
• facebookresearch / SelfCite
  Code for the ICML 2025 paper "SelfCite Self-Supervised Alignment for Context Attribution in Large Language Models"
  ☆21Updated 3 weeks ago
• epfl-dlab / llm-latent-language
  Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".
  ☆80Updated last year