Fast and customizable text tokenization library with BPE and SentencePiece support
☆330Jan 10, 2026Updated last month
Alternatives and similar repositories for Tokenizer
Users that are interested in Tokenizer are comparing it to the libraries listed below
Sorting:
- Lightweight C++ translator for OpenNMT Torch models (deprecated)☆81Apr 7, 2020Updated 5 years ago
- A Comprehensive survey on business use cases of AI that help them thrive in the digital economy☆13Oct 7, 2020Updated 5 years ago
- This is a AUTOSAR documents specific retriever based on LLM and RAG.☆16Nov 12, 2024Updated last year
- Library and command line utility to do approximate string matching of a source against a bitext index and get matched source and target.☆51Apr 22, 2025Updated 10 months ago
- Neural machine translation and sequence learning using TensorFlow☆1,485Oct 14, 2023Updated 2 years ago
- A library for data streaming and augmentation☆21May 5, 2025Updated 9 months ago
- An experimental desktop client for using Claude Desktop's MCP with Novelcrafter codices.☆10Dec 3, 2024Updated last year
- Consider is a parser for the ThinkGear protocol used by NeuroSky devices (MindSet, BrainBand and others).☆16Apr 3, 2012Updated 13 years ago
- Android TextMate Bundle☆17Mar 20, 2009Updated 16 years ago
- Citation Manager for OJS☆13Jun 4, 2024Updated last year
- ☆11Dec 10, 2022Updated 3 years ago
- A simple node.js wrapper for Stanford CoreNLP.☆10Aug 7, 2014Updated 11 years ago
- A proselint linter for use with Phabricator's arc command line tool.☆17Jun 17, 2016Updated 9 years ago
- Useful collection of webrat Textmate snippets meant for use with the RSpec Story and/or Cucumber bundles☆79Aug 7, 2009Updated 16 years ago
- Hello world demonstration for Weblate☆14Jan 20, 2026Updated last month
- CROMER (CROss-document Main Events and entities Recognition), is a tool for cross-document coreference☆12Jan 14, 2015Updated 11 years ago
- The OpenCitations metadata model: documents and other material.☆19Nov 20, 2025Updated 3 months ago
- ☆17Aug 28, 2025Updated 6 months ago
- Exploration: using technology to aid people who lack both the ability to speak and fine motor control.☆22Oct 24, 2024Updated last year
- A bot to add citation data from OpenCitations to Wikidata☆12May 23, 2023Updated 2 years ago
- 一个纯实验项目☆11Sep 13, 2011Updated 14 years ago
- A crawler which uses regular expression to catch data from website.☆48Feb 6, 2010Updated 16 years ago
- Fast Neural Machine Translation in C++☆1,421Aug 25, 2023Updated 2 years ago
- Open Source Neural Machine Translation and (Large) Language Models in PyTorch☆6,989Oct 14, 2025Updated 4 months ago
- Fast inference engine for Transformer models☆4,326Feb 4, 2026Updated 3 weeks ago
- Scripts to preprocess training and test data and to run fast_align and giza☆107Nov 2, 2021Updated 4 years ago
- Simple, fast unsupervised word aligner☆767Jul 19, 2022Updated 3 years ago
- Fast BPE☆679Jun 18, 2024Updated last year
- We release a dataset based on Wikipedia sentences and the corresponding translations in 6 different languages along with the scores (scal…☆81Aug 31, 2021Updated 4 years ago
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,668Feb 22, 2026Updated last week
- Physioboard demonstrates iOS/Python/Web prototypes for "real-time" physiological data acquisition and visualization (i.e., skin conductan…☆17Apr 28, 2013Updated 12 years ago
- Ruby binding for the igraph library.☆33Aug 13, 2009Updated 16 years ago
- 通识教育的信息、系统论、控制论解读☆12Jan 16, 2019Updated 7 years ago
- ☆13May 10, 2023Updated 2 years ago
- A repository of Juris-M style modules☆16Jan 17, 2024Updated 2 years ago
- A Sublime Text 3 client for Vale Server.☆13Dec 7, 2020Updated 5 years ago
- This GUI aims to simplify the process of converting GGUF files to llamafile format by providing an intuitive and convenient way for users…☆14Jan 2, 2026Updated last month
- Domain-specific language for mobile (web) applications☆16May 12, 2010Updated 15 years ago
- Textmate bundle for RSpec.☆200Jul 6, 2010Updated 15 years ago