1kkiRen / Tokenizer-ChangerLinks
Python script for manipulating the existing tokenizer.
☆21Updated 3 weeks ago
Alternatives and similar repositories for Tokenizer-Changer
Users that are interested in Tokenizer-Changer are comparing it to the libraries listed below
Sorting:
- Code for Zero-Shot Tokenizer Transfer☆142Updated 11 months ago
- Suri: Multi-constraint instruction following for long-form text generation (EMNLP’24)☆27Updated 3 months ago
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"☆121Updated last year
- LongMIT: Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets☆39Updated last year
- Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible…☆92Updated 3 weeks ago
- Code for ICML 25 paper "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"☆48Updated 6 months ago
- Code for paper "Patch-Level Training for Large Language Models"☆96Updated last month
- [NeurIPS 2023] Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective☆39Updated 2 years ago
- Official PyTorch Implementation of EMoE: Unlocking Emergent Modularity in Large Language Models [main conference @ NAACL2024]☆38Updated last year
- Official repository for ACL 2025 paper "Model Extrapolation Expedites Alignment"☆76Updated 7 months ago
- ☆12Updated last year
- Long Context Extension and Generalization in LLMs☆62Updated last year
- SiLLM is a Simultaneous Machine Translation (SiMT) Framework. It utilizes a Large Language model as the translation model and employs a t…☆17Updated last year
- A collection of instruction data and scripts for machine translation.☆20Updated 2 years ago
- ☆35Updated 2 years ago
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆78Updated last year
- Longitudinal Evaluation of LLMs via Data Compression☆33Updated last year
- ☆72Updated last year
- WorldSense benchmark for grounded reasoning in language models☆22Updated 2 years ago
- ☆44Updated last year
- Evaluation results for Machine Translation within the BigScience project☆11Updated 2 years ago
- SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects☆23Updated 11 months ago
- Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023☆107Updated last year
- TrustJudge is a probabilistic evaluation framework that reduces score-comparison and pairwise transitivity inconsistencies in LLM-as-a-ju…☆38Updated 3 months ago
- ☆85Updated last month
- Nano repo for RL training of LLMs☆70Updated 2 months ago
- Evaluate your agent memory on real-world dialogues, not LLM-simulated dialogues.☆35Updated 6 months ago
- Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"☆29Updated last year
- Code for the ICML 2025 paper "SelfCite Self-Supervised Alignment for Context Attribution in Large Language Models"☆21Updated 3 weeks ago
- Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".☆80Updated last year