🍡 30x faster tokenization for every HuggingFace model
☆47May 28, 2026Updated last month
Alternatives and similar repositories for tokie
Users that are interested in tokie are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- FlexiTokens☆23Dec 27, 2025Updated 6 months ago
- Next-generation Punkt sentence boundary detection with zero dependencies☆32Nov 18, 2025Updated 7 months ago
- High performance implementation of the WARP (SIGIR'25) retrieval engine.☆35May 21, 2026Updated last month
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆40Sep 20, 2025Updated 9 months ago
- Official repository of the Seismic library.☆135Jul 6, 2026Updated last week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Layerwise Relevance Visualization in Convolutional Text Graph Classifiers☆11Jun 2, 2021Updated 5 years ago
- Official repository of TACHIOM.☆62Jun 26, 2026Updated 2 weeks ago
- Trainable embedding transformation for confidence estimation, feature extraction, explainability and conversion from dense to sparse.☆28Jun 23, 2026Updated 3 weeks ago
- ANE accelerated embedding models!☆20Dec 11, 2024Updated last year
- Combining encoder-based language models☆11Nov 11, 2021Updated 4 years ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 3 years ago
- ☆15Oct 19, 2020Updated 5 years ago
- [ACL 20] Probing Linguistic Features of Sentence-level Representations in Neural Relation Extraction☆13Apr 21, 2020Updated 6 years ago
- NLP with Rust for Python 🦀🐍☆72Jun 9, 2026Updated last month
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Python Module implementing SRP☆12Jul 29, 2022Updated 3 years ago
- User-friendly viewer for Parquet files☆16May 8, 2026Updated 2 months ago
- Efficient BM25 with DuckDB 🦆☆68Dec 20, 2024Updated last year
- https://footprints.baulab.info☆17Oct 4, 2024Updated last year
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.☆69Jul 6, 2025Updated last year
- A massively multilingual modern encoder language model☆145Jan 20, 2026Updated 5 months ago
- ☆13Nov 15, 2017Updated 8 years ago
- Personal README for Github profile.☆10Jun 12, 2023Updated 3 years ago
- allennlp + streamlit demo☆21Oct 2, 2019Updated 6 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- RelEx - A simple framework for Relation Extraction built on AllenNLP☆15Jun 17, 2020Updated 6 years ago
- Python implementation of Levenshtein distance and Levenshtein automata matching☆27May 8, 2019Updated 7 years ago
- ☆57Jul 10, 2025Updated last year
- Learning BPE embeddings by first learning a segmentation model and then training word2vec☆19Dec 18, 2022Updated 3 years ago
- Code for "Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations" [NAACL Findings 2024]☆14Apr 3, 2026Updated 3 months ago
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆46Jul 4, 2026Updated last week
- Demo code for Gemini Live Integration☆13Jul 29, 2025Updated 11 months ago
- Implementation of Nested Named Entity Recognition using Flair☆24Oct 29, 2021Updated 4 years ago
- benchmarks for LLM tokenizers☆20Mar 25, 2026Updated 3 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Page of the course "Information Retrieval" at Department of Computer Science, University of Pisa☆21Dec 18, 2025Updated 6 months ago
- Robust and fast topic models with sentence-transformers.☆118Updated this week
- The Referential Reader: A Recurrent Entity Network for Anaphora Resolution, published at ACL 2019☆20Nov 1, 2019Updated 6 years ago
- Discord bot that knows a lot about Pokemons :)☆14Dec 25, 2024Updated last year
- Automatically exported from code.google.com/p/transducersaurus☆11Apr 1, 2015Updated 11 years ago
- ☆57Dec 27, 2025Updated 6 months ago
- Nanoloop source files for the album "Prime 16"☆11Mar 7, 2026Updated 4 months ago