This is the repository for MorphScore, a tokenizer evaluation framework for morphological alignment.
☆19Jul 10, 2025Updated last year
Alternatives and similar repositories for morphscore
Users that are interested in morphscore are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆58Nov 6, 2026Updated last week
- Simple-to-use scoring function for arbitrarily tokenized texts.☆52Oct 18, 2026Updated last month
- A tool to calculate dataset scaling needed to achieve cross-lingual training data equity.☆13Aug 14, 2024Updated 2 years ago
- PathPiece tokenizer☆14Nov 10, 2024Updated 2 years ago
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆27Nov 11, 2026Updated last week
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆11Jan 27, 2025Updated last year
- Source code for paper Grammatical Error Correction in Low-Resource Scenarios (W-NUT 2019)☆13Jun 21, 2022Updated 4 years ago
- Metric Space Magnitude Computations☆15Jun 30, 2026Updated 4 months ago
- Materials for "Natural Language Processing for Multilingual Task-Oriented Dialogue" Tutorial at ACL 2022☆14May 21, 2022Updated 4 years ago
- LTG-Bert☆35Jan 8, 2024Updated 2 years ago
- An initiative to collect and distribute resources for co-reference resolution in a unified standard.☆28May 12, 2024Updated 2 years ago
- Code for SaGe subword tokenizer (EACL 2023)☆28Nov 30, 2024Updated last year
- Official repository for the EMNLP 2019 paper, "Learning Invariant Representations of Social Media Users."☆12Aug 27, 2021Updated 5 years ago
- Type-and-example directed program synthesis using example propagation, as described in Program Synthesis Using Example Propagation.☆16Jul 26, 2026Updated 3 months ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- ☆27Aug 19, 2025Updated last year
- Code for "Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding" (EMNLP 2020).☆11May 1, 2025Updated last year
- ☆11Nov 8, 2023Updated 3 years ago
- Research code for the paper "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models"☆27Oct 3, 2021Updated 5 years ago
- ☆12Apr 19, 2022Updated 4 years ago
- ☆15Aug 2, 2024Updated 2 years ago
- HELP: a dataset for Handling Entailments with Lexical and logical Phenomena (Ver.1.0)☆15Jul 20, 2023Updated 3 years ago
- Fonts code (http://ilab.usc.edu/datasets/fonts)☆11Apr 14, 2021Updated 5 years ago
- Official code release for "SuperBPE: Space Travel for Language Models"☆102May 28, 2026Updated 5 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Calculates bounds on the sofa moving problem☆14Sep 12, 2019Updated 7 years ago
- [ACL 2025] TokAlign: Efficient Vocabulary Adaptation via Token Alignment☆12Jun 4, 2025Updated last year
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Nov 5, 2026Updated 2 weeks ago
- A Multilingual Keyboard Layout-Based Typo Generator☆19Nov 23, 2025Updated 11 months ago
- A list of resources dedicated to compositionality☆14Feb 21, 2019Updated 7 years ago
- Official repository of the ACL 2024 paper "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!".☆10Nov 19, 2024Updated 2 years ago
- Official repository for the paper: Can Large Language Models Capture Human Annotator Disagreements?☆13Jul 27, 2025Updated last year
- Split bib files for anthology bibliography for overleaf☆11Aug 25, 2024Updated 2 years ago
- A benchmark with locally sourced multilingual questions for 31 languages.☆19Updated this week
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- ☆37Jul 27, 2025Updated last year
- Basic Arabic shaping for RoboFont 3.4☆10Jan 12, 2026Updated 10 months ago
- ☆15Jul 7, 2026Updated 4 months ago
- Tools for calculating psycholinguistically-relevant metrics of language statistics using transformer language models☆13Nov 11, 2022Updated 4 years ago
- ACL 2021 paper "Style is NOT a single variable: Case Studies for Cross-Style Language Understanding " by Dongyeop Kang and Eduard Hovy☆15Jul 19, 2021Updated 5 years ago
- Language modeling that treats text as images, leveraging visual structure for enhanced understanding.☆15Oct 22, 2026Updated 3 weeks ago
- Can Large Language Models Identify Authorship? (EMNLP 2024 Findings)☆14Feb 4, 2025Updated last year
Are these results useful?