A curated list of tokenizer libraries for blazing-fast NLP processing.
☆14Apr 10, 2025Updated last year
Alternatives and similar repositories for awesome-tokenizers
Users that are interested in awesome-tokenizers are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Lossless normalization of uppercase characters: Go, C++ & JavaScript☆11Jul 7, 2026Updated 3 months ago
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆58Aug 31, 2026Updated last month
- 2D chess pieces inspired by handmade wooden chess sets, featuring cuteness and simplicity☆15Feb 14, 2026Updated 8 months ago
- A code snippet that proves that there is no legal, potentially non-reachable chess position with more than 218 moves.☆23May 23, 2024Updated 2 years ago
- FlexiTokens☆23Dec 27, 2025Updated 9 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- SAGA - Phonetic transcription software for all Spanish variants.☆13Nov 12, 2020Updated 5 years ago
- SMTP Server Tester, by Alton Johnson (alton.jx@gmail.com)☆20Jul 3, 2015Updated 11 years ago
- CRIM Intervals is a Python library for analysis of encoded scores. Based on Music21, it makes short work of finding melodic, harmonic, a…☆22Aug 18, 2026Updated 2 months ago
- windows setup script☆11Jan 22, 2023Updated 3 years ago
- Human-Aligned Chess With a Bit of Search☆24Oct 4, 2026Updated 2 weeks ago
- ☆12Sep 29, 2017Updated 9 years ago
- The source of the phonetic transcriptions is Oxford Advanced Learner's Dictionary (3rd ed.), available from the Oxford Text Archive (http…☆30Apr 25, 2017Updated 9 years ago
- Aix: Efficiently storing and querying chess game collections☆29Mar 29, 2026Updated 6 months ago
- text preprocessing library with framework for composable tokenizations☆13Jun 15, 2024Updated 2 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- AAAI 2022 Paper: Bet even Beth Harmon couldn't learn chess like that :)☆39Mar 3, 2021Updated 5 years ago
- ☆10Apr 26, 2021Updated 5 years ago
- Fully indexable dictionaries - access/rank/select operations☆13Sep 10, 2018Updated 8 years ago
- A collection of Pythonic subword tokenisers and text preprocessing tools.☆15Updated this week
- Notebooks and other course materials for Emory QTM 340 (Fall 2022)☆12Dec 13, 2022Updated 3 years ago
- Simple currying in julia☆14Nov 20, 2019Updated 6 years ago
- Inline PTX Assembly in CUDA example☆15Oct 5, 2026Updated 2 weeks ago
- 🍡 30x faster tokenization for every HuggingFace model☆55Updated this week
- Demo of all Epics Unreal AI Navigation features☆10Jul 31, 2024Updated 2 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- A music21-extending set of converters (Humdrum, MEI, and ABC readers and writers), as well as a CLI app to convert music notation file fo…☆32Oct 8, 2026Updated 2 weeks ago
- Benchmark of common hash functions☆10Sep 15, 2019Updated 7 years ago
- Train to 94% on CIFAR-10 in 4.4 seconds on a single A100☆12Dec 30, 2023Updated 2 years ago
- Fast Travel system similar to Assassin's Creed Odyssey☆11Feb 26, 2021Updated 5 years ago
- aabbyy training☆10Jun 7, 2020Updated 6 years ago
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025)☆11Jul 15, 2025Updated last year
- Adds toon shader and outline to selected object☆11Feb 18, 2023Updated 3 years ago
- Adds meshes, lights etc., scaled to custom dimensions☆11Jul 1, 2021Updated 5 years ago
- ☆10Feb 2, 2023Updated 3 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Simple-to-use scoring function for arbitrarily tokenized texts.☆52Feb 19, 2025Updated last year
- Experiments in visualizing the local cosmic web☆30Apr 6, 2026Updated 6 months ago
- Open Hardware Controller for Blender (contains KiCAD, Gerber, CircuitPython, and Addon files)☆13Feb 23, 2023Updated 3 years ago
- ☆15Oct 24, 2023Updated 2 years ago
- ☆11Jan 27, 2025Updated last year
- python to tax form compiler☆13Jun 6, 2022Updated 4 years ago
- download, compile && prepare cpp-ethereum dependencies for windows☆17Jan 14, 2019Updated 7 years ago