Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"
☆15Sep 2, 2026Updated 2 months ago
Alternatives and similar repositories for tokenizers_intrinsic_benchmark
Users that are interested in tokenizers_intrinsic_benchmark are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆11Aug 1, 2019Updated 7 years ago
- ☆12Apr 24, 2025Updated last year
- This is the repository for MorphScore, a tokenizer evaluation framework for morphological alignment.☆19Jul 10, 2025Updated last year
- Simple-to-use scoring function for arbitrarily tokenized texts.☆52Oct 18, 2026Updated 2 weeks ago
- ☆11Jan 28, 2026Updated 9 months ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- ☆18Feb 1, 2023Updated 3 years ago
- ☆15Jul 7, 2026Updated 3 months ago
- Code for "Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding" (EMNLP 2020).☆11May 1, 2025Updated last year
- Official repository for the EMNLP 2019 paper, "Learning Invariant Representations of Social Media Users."☆12Aug 27, 2021Updated 5 years ago
- Type-and-example directed program synthesis using example propagation, as described in Program Synthesis Using Example Propagation.☆16Jul 26, 2026Updated 3 months ago
- Avalinguo Audio Dataset: Dataset for Speaker Fluency Level Classification☆13Aug 13, 2018Updated 8 years ago
- ☆10Nov 8, 2023Updated 2 years ago
- Demo server for TREC LiveQA competition☆11Dec 7, 2016Updated 9 years ago
- Finds snippets in iambic pentameter in English-language text and tries to combine them to a rhyming sonnet.☆13Jan 5, 2023Updated 3 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- PANiC - PAraphrasing Noun-Compounds☆15Apr 6, 2018Updated 8 years ago
- 🌍 A simple script for taking automated screenshots from a Leaflet map☆16Mar 29, 2018Updated 8 years ago
- Code for the ILNewsDiff Twitter account☆10May 23, 2023Updated 3 years ago
- Official implementation of "Data Mixture Inference: What do BPE tokenizers reveal about their training data?"☆22May 15, 2025Updated last year
- A tool to calculate dataset scaling needed to achieve cross-lingual training data equity.☆13Aug 14, 2024Updated 2 years ago
- Can Large Language Models Identify Authorship? (EMNLP 2024 Findings)☆14Feb 4, 2025Updated last year
- Metric Space Magnitude Computations☆15Jun 30, 2026Updated 4 months ago
- ACL 2021 paper "Style is NOT a single variable: Case Studies for Cross-Style Language Understanding " by Dongyeop Kang and Eduard Hovy☆15Jul 19, 2021Updated 5 years ago
- Tools for calculating psycholinguistically-relevant metrics of language statistics using transformer language models☆13Nov 11, 2022Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Statistics on multilingual datasets☆17Jul 12, 2022Updated 4 years ago
- Complete set of English dialect transformation rules and evaluation code☆16Jun 7, 2024Updated 2 years ago
- Code for "Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations" [NAACL Findings 2024]☆14Apr 3, 2026Updated 7 months ago
- MorphyNet: a Large Multilingual Database of Derivational and Inflectional Morphology (+morpheme segmentation)☆64Apr 2, 2023Updated 3 years ago
- Dialogue Act classification☆18Jan 15, 2024Updated 2 years ago
- Materials for LOT School 2023, "Language Learning: A Data-Driven Approach"☆14Aug 14, 2024Updated 2 years ago
- Helpful WebUI for Senior Area Chair of ACL Rolling Review (ARR)☆17Aug 11, 2026Updated 2 months ago
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Updated this week
- Typst linguistic examples with minimalist syntax☆19Oct 20, 2026Updated 2 weeks ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Jednoduchá implementace platební brány GP webpay v PHP☆13Aug 1, 2011Updated 15 years ago
- Nanoloop source files for the album "Prime 16"☆12Mar 7, 2026Updated 8 months ago
- ☆18Feb 4, 2025Updated last year
- A Neural Attention Model for Abstractive Sentence Summarization in DyNet☆19May 3, 2018Updated 8 years ago
- ☆13Apr 16, 2021Updated 5 years ago
- ☆103Jul 4, 2025Updated last year
- Official code release for "SuperBPE: Space Travel for Language Models"☆101May 28, 2026Updated 5 months ago