benchmarks for LLM tokenizers
☆20Mar 25, 2026Updated 4 months ago
Alternatives and similar repositories for tokka-bench
Users that are interested in tokka-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 🍡 30x faster tokenization for every HuggingFace model☆49Updated this week
- ☆13Nov 15, 2023Updated 2 years ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 3 years ago
- Calculate the maximum string length for a photovoltaic system☆12Dec 8, 2022Updated 3 years ago
- A library to handle time series in NumPy/SciPy☆16Aug 7, 2010Updated 15 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆46Jul 4, 2026Updated 3 weeks ago
- Sequencable robot operated digital tape machine☆18May 14, 2024Updated 2 years ago
- An English lexical database from the Big 🍎, let's go Mets baby love da Mets☆18Updated this week
- Angular feature flag library with Unleash integration, provides reactive programing, type-safe feature flag management with zero-configur…☆33Updated this week
- Calculate parameters that are not available in datasheet of photovoltaic modules using equations derived from diode model☆13Mar 9, 2017Updated 9 years ago
- Floating DevTools panel that intercepts OpenAI, Anthropic, Gemini, Mistral, Groq, and Cohere calls in your Angular app and shows you the …☆155Jun 19, 2026Updated last month
- Python implementation of the control charts used for process monitoring and anomaly detection☆13Jun 18, 2023Updated 3 years ago
- OCF's DataPipe based dataloader for training and inference☆15Jun 2, 2025Updated last year
- ☆10Mar 18, 2021Updated 5 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A Combinatory Categorial Grammar library.☆22Nov 11, 2013Updated 12 years ago
- TThe program will take as input - code, algorithm, workflow or manual and will give as output- a flowchart for it.☆17Jul 27, 2024Updated 2 years ago
- pyndl implements a Naive discriminative learning which is a learning and classification models based on the Rescorla-Wagner equations in …☆13Dec 8, 2025Updated 7 months ago
- Chrome extension which lets you watch WWDC Developer Videos in Google Chrome (thus not having to use Safari)☆17Oct 11, 2017Updated 8 years ago
- ☆18Feb 1, 2023Updated 3 years ago
- Time series analytics with Python☆17Dec 7, 2015Updated 10 years ago
- UMLS in Python with MongoDB.☆18Dec 7, 2018Updated 7 years ago
- ☆14Feb 9, 2022Updated 4 years ago
- ☆10Jun 9, 2023Updated 3 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Tools for working with PubMed data.☆15Sep 26, 2020Updated 5 years ago
- Basis data, analysis, and visualization☆15Jan 3, 2014Updated 12 years ago
- ☆29Jun 23, 2022Updated 4 years ago
- ShopEasy is a containerized e‑commerce boilerplate with an Angular SPA and Spring Boot API (JWT‑secure), backed by MySQL and Kafka, deplo…☆17Sep 5, 2025Updated 10 months ago
- This repository hosts a ChatGPT plugin that enables the uploading of a PDF URL. The plugin extracts information from the PDF, to enhance …☆11May 24, 2023Updated 3 years ago
- Python package that offers text scrubbing functionality, providing building blocks for string cleaning as well as normalizing geographica…☆22Aug 26, 2024Updated last year
- Acoustic distance measure for comparing pronunciations☆17Aug 2, 2022Updated 3 years ago
- FlexiTokens☆23Dec 27, 2025Updated 7 months ago
- ☆12Aug 1, 2025Updated 11 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Code and Data Repo for the CoNLL Paper -- Future Lens: Anticipating Subsequent Tokens from a Single Hidden State☆21Oct 24, 2025Updated 9 months ago
- Dataset accompanying AI Rules? Characterizing Reddit Community Policies Towards AI-Generated Content☆16Apr 29, 2025Updated last year
- MUltiband Satellite Imagery for object Classification (MUSIC) to detect Photovoltaic Power Plants☆17Jan 26, 2018Updated 8 years ago
- A dependencies analyzer for Python☆31Dec 4, 2017Updated 8 years ago
- ☆20Feb 17, 2023Updated 3 years ago
- Controllable Language Model Interactions in TypeScript☆10May 17, 2024Updated 2 years ago
- Evolutionary Search for expert-level performance on any task with environmental feedback☆14Oct 12, 2025Updated 9 months ago