Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"
β22Feb 14, 2024Updated 2 years ago
Alternatives and similar repositories for tokenizer-bench
Users that are interested in tokenizer-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 0-Shot Tokenizer Transplantβ14May 16, 2025Updated last year
- The evalution suite for the π Open Portuguese LLM Leaderboardβ25Aug 31, 2025Updated 11 months ago
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGramβ19Updated this week
- Master thesis: Exploring bias in German NLG (GPT-3 & GerPT-2). Applies regard classification and bias mitigation triggers.β16Sep 25, 2024Updated last year
- β21Jun 4, 2024Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer β’ AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Temporary remove unused tokens during training to save ram and speed.β23Jun 15, 2025Updated last year
- Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [ACL 2026]β20Apr 18, 2026Updated 3 months ago
- Ukrainian ELECTRA modelβ12Mar 11, 2023Updated 3 years ago
- β16May 14, 2024Updated 2 years ago
- [ACL 2025] π Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentβ11Apr 6, 2025Updated last year
- Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningβ30Jan 25, 2023Updated 3 years ago
- β10Nov 8, 2023Updated 2 years ago
- A scalable benchmark for state representation learning in visual reinforcement learning.β17Jun 23, 2025Updated last year
- Implementation of Cascaded Head-colliding Attention (ACL'2021)β11Sep 16, 2021Updated 4 years ago
- End-to-end encrypted email - Proton Mail β’ AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Code for SaGe subword tokenizer (EACL 2023)β28Nov 30, 2024Updated last year
- [ICML 2026] Improving GPT via a simple normalization strategyβ15May 22, 2026Updated 2 months ago
- Minimal code to train ELMo models in recent versions of TensorFlowβ14Jun 16, 2026Updated last month
- Getting interpretable dimensions in word embedding spaces.β15Jul 6, 2023Updated 3 years ago
- Resources related to EMNLP 2021 paper "FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations"β13Dec 14, 2021Updated 4 years ago
- My personal websiteβ11May 30, 2026Updated 2 months ago
- AI-Powered Thesis Review Toolβ17Aug 8, 2025Updated last year
- Plancraft is a minecraft environment and agent suite to test planning capabilities in LLMsβ31Nov 7, 2025Updated 9 months ago
- Implementation of the paper 'Sentence Bottleneck Autoencoders from Transformer Language Models'β17Mar 14, 2022Updated 4 years ago
- End-to-end encrypted email - Proton Mail β’ AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- [NeurIPS 2025] MergeBench: A Benchmark for Merging Domain-Specialized LLMsβ49Feb 11, 2026Updated 6 months ago
- suffix array construction and searching algorithms for in-memory binary data.β13Sep 10, 2022Updated 3 years ago
- A byte-level decoder architecture that matches the performance of tokenized Transformers.β68Apr 24, 2024Updated 2 years ago
- From Hero to ZΓ©roe: A Benchmark of Low-Level Adversarial Attacksβ15Feb 23, 2023Updated 3 years ago
- βοΈ Sentence segmentation with wtpsplit's state-of-the-art Segment any Text (SaT) modelsβ40May 2, 2026Updated 3 months ago
- Python module to remove wiki markup text.β10Jan 15, 2016Updated 10 years ago
- Code for "Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations" [NAACL Findings 2024]β14Apr 3, 2026Updated 4 months ago
- Set-Equivariant Deep Learning Modelsβ22Dec 23, 2021Updated 4 years ago
- Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"β15Nov 26, 2024Updated last year
- Deploy on Railway without the complexity - Free Credits Offer β’ AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- KDSS is the framework for knowledge distillation from LLMsβ12Nov 5, 2025Updated 9 months ago
- [ποΈ1λ±(μ₯κ΄μ) μ루μ ] 2022 κ΅λ¦½κ΅μ΄μ μΈκ³΅ μ§λ₯ μΈμ΄ λ₯λ ₯ νκ° (μΌνλͺ° 리뷰 λ°μ΄ν° μμ± κΈ°λ° κ°μ± λΆμ : Aspect-Based Sentiment Analysis)β11Jun 6, 2023Updated 3 years ago
- A Python package to wrap a compiled soccer-twos reinforcement learning environment with gym-compatible interfaces.β28Feb 25, 2023Updated 3 years ago
- A tool for extracting plain text from Wikipedia dumpsβ15Sep 13, 2018Updated 7 years ago
- Example for a Monty-enabled RLM in DSPyβ20Feb 16, 2026Updated 5 months ago
- Micro-framework for publishing linked dataβ11Aug 1, 2017Updated 9 years ago
- Find informative examples to efficiently (human)-evaluate NLG models.β17Apr 22, 2026Updated 3 months ago