A curated list of tokenizer libraries for blazing-fast NLP processing.
☆14Apr 10, 2025Updated last year
Alternatives and similar repositories for awesome-tokenizers
Users that are interested in awesome-tokenizers are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- The Lichess puzzle database combined with game information.☆12Jun 1, 2026Updated 5 months ago
- Music of Gaspar van Weerbeke as Humdrum digital scores.☆10Aug 28, 2022Updated 4 years ago
- Anagram Generator in Python☆10Jul 20, 2021Updated 5 years ago
- Lossless normalization of uppercase characters: Go, C++ & JavaScript☆12Jul 7, 2026Updated 4 months ago
- 2D chess pieces inspired by handmade wooden chess sets, featuring cuteness and simplicity☆15Feb 14, 2026Updated 9 months ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆58Nov 6, 2026Updated last week
- Scores for The 1520s Project, a digital library of European polyphonic music, ca. 1510 to ca. 1540. It seeks to address how, when, and wh…☆12Nov 4, 2026Updated last week
- A code snippet that proves that there is no legal, potentially non-reachable chess position with more than 218 moves.☆23May 23, 2024Updated 2 years ago
- FlexiTokens☆23Dec 27, 2025Updated 10 months ago
- SAGA - Phonetic transcription software for all Spanish variants.☆13Nov 12, 2020Updated 6 years ago
- SMTP Server Tester, by Alton Johnson (alton.jx@gmail.com)☆20Jul 3, 2015Updated 11 years ago
- NanoGPT (124M) in 5 minutes☆18Feb 14, 2025Updated last year
- windows setup script☆11Jan 22, 2023Updated 3 years ago
- ☆12Sep 29, 2017Updated 9 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Human-Aligned Chess With a Bit of Search☆24Oct 4, 2026Updated last month
- CRIM Intervals is a Python library for analysis of encoded scores. Based on Music21, it makes short work of finding melodic, harmonic, a…☆22Updated this week
- Solve anagrams from the command line☆17Aug 8, 2019Updated 7 years ago
- The source of the phonetic transcriptions is Oxford Advanced Learner's Dictionary (3rd ed.), available from the Oxford Text Archive (http…☆30Apr 25, 2017Updated 9 years ago
- AAAI 2022 Paper: Bet even Beth Harmon couldn't learn chess like that :)☆39Mar 3, 2021Updated 5 years ago
- Fully indexable dictionaries - access/rank/select operations☆13Sep 10, 2018Updated 8 years ago
- ☆10Apr 26, 2021Updated 5 years ago
- Aix: Efficiently storing and querying chess game collections☆31Updated this week
- text preprocessing library with framework for composable tokenizations☆13Jun 15, 2024Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Notebooks and other course materials for Emory QTM 340 (Fall 2022)☆12Dec 13, 2022Updated 3 years ago
- Simple currying in julia☆14Nov 20, 2019Updated 6 years ago
- Inline PTX Assembly in CUDA example☆15Oct 5, 2026Updated last month
- 🍡 30x faster tokenization for every HuggingFace model☆57Updated this week
- A collection of Pythonic subword tokenisers and text preprocessing tools.☆15Updated this week
- Train to 94% on CIFAR-10 in 4.4 seconds on a single A100☆12Dec 30, 2023Updated 2 years ago
- Demo of all Epics Unreal AI Navigation features☆10Jul 31, 2024Updated 2 years ago
- Digital scores for all composers in the Josquin Research Project, which focuses on vocal music, ca. 1420–1520.☆26Updated this week
- Fast Travel system similar to Assassin's Creed Odyssey☆11Feb 26, 2021Updated 5 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- aabbyy training☆10Jun 7, 2020Updated 6 years ago
- ☆10Feb 2, 2023Updated 3 years ago
- Experiments in visualizing the local cosmic web☆30Apr 6, 2026Updated 7 months ago
- Adds meshes, lights etc., scaled to custom dimensions☆11Jul 1, 2021Updated 5 years ago
- Open Hardware Controller for Blender (contains KiCAD, Gerber, CircuitPython, and Addon files)☆13Feb 23, 2023Updated 3 years ago
- Raft Consensus Algorithm implementation in Zig☆13Nov 24, 2025Updated 11 months ago
- download, compile && prepare cpp-ethereum dependencies for windows☆17Jan 14, 2019Updated 7 years ago
Are these results useful?