benchmarks for LLM tokenizers
☆21Mar 25, 2026Updated 6 months ago
Alternatives and similar repositories for tokka-bench
Users that are interested in tokka-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 🍡 30x faster tokenization for every HuggingFace model☆55Oct 9, 2026Updated last week
- Material for the series of seminars on Large Language Models☆33Apr 21, 2024Updated 2 years ago
- Automatically exported from code.google.com/p/transducersaurus☆11Apr 1, 2015Updated 11 years ago
- Nanoloop source files for the album "Prime 16"☆12Mar 7, 2026Updated 7 months ago
- Python SDK for the Solcast API☆12Oct 1, 2026Updated 2 weeks ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- ☆12Sep 1, 2021Updated 5 years ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 3 years ago
- Python implementation of DBCLASD, a non-parametric clustering algorithm☆12May 28, 2020Updated 6 years ago
- A library to handle time series in NumPy/SciPy☆16Aug 7, 2010Updated 16 years ago
- Tokenization across languages. Useful as preprocessing for subword tokenization.☆19Feb 7, 2023Updated 3 years ago
- A Python library for summarizing event data into multivariate categorical data☆10May 16, 2020Updated 6 years ago
- Amiga DF0: / DF1: (_SEL0/_SEL1) internal automatic drive switch using Arduino Pro Mini and minimal hardware.☆12Mar 2, 2018Updated 8 years ago
- Calculate parameters that are not available in datasheet of photovoltaic modules using equations derived from diode model☆13Mar 9, 2017Updated 9 years ago
- The CODWOE shared task invites you to compare two types of semantic descriptions: dictionary glosses and word embedding representations. …☆12Jul 13, 2022Updated 4 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Floating DevTools panel that intercepts OpenAI, Anthropic, Gemini, Mistral, Groq, and Cohere calls in your Angular app and shows you the …☆225Jun 19, 2026Updated 3 months ago
- Python implementation of the control charts used for process monitoring and anomaly detection☆13Jun 18, 2023Updated 3 years ago
- Deep CCA (DCCA) network☆15Apr 12, 2016Updated 10 years ago
- OCF's DataPipe based dataloader for training and inference☆15Jun 2, 2025Updated last year
- ☆10Mar 18, 2021Updated 5 years ago
- pyndl implements a Naive discriminative learning which is a learning and classification models based on the Rescorla-Wagner equations in …☆13Dec 8, 2025Updated 10 months ago
- Time series analytics with Python☆17Dec 7, 2015Updated 10 years ago
- UMLS in Python with MongoDB.☆18Aug 5, 2026Updated 2 months ago
- Code repo for Spatio-Temporal Denoising Graph Autoencoder (STD-GAE)☆12Sep 6, 2022Updated 4 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- ☆14Feb 9, 2022Updated 4 years ago
- Tools for working with PubMed data.☆15Sep 26, 2020Updated 6 years ago
- Basis data, analysis, and visualization☆15Jan 3, 2014Updated 12 years ago
- Amibian - Your device, your Amiga.☆16Jan 25, 2018Updated 8 years ago
- ShopEasy is a containerized e‑commerce boilerplate with an Angular SPA and Spring Boot API (JWT‑secure), backed by MySQL and Kafka, deplo…☆15Sep 5, 2025Updated last year
- Learn how to use Redash to query and display your database data☆24Jan 16, 2018Updated 8 years ago
- This repository hosts a ChatGPT plugin that enables the uploading of a PDF URL. The plugin extracts information from the PDF, to enhance …☆10May 24, 2023Updated 3 years ago
- ☆12Aug 1, 2025Updated last year
- CLI tool for checking contribution from git repository☆16Mar 30, 2022Updated 4 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Dataset accompanying AI Rules? Characterizing Reddit Community Policies Towards AI-Generated Content☆16Apr 29, 2025Updated last year
- ☆35Updated this week
- A dependencies analyzer for Python☆30Dec 4, 2017Updated 8 years ago
- Controllable Language Model Interactions in TypeScript☆10May 17, 2024Updated 2 years ago
- Evolutionary Search for expert-level performance on any task with environmental feedback☆14Oct 12, 2025Updated last year
- Automated cloud masking for Landsat MSS images☆18Dec 12, 2018Updated 7 years ago
- Jupyter notebooks for gaining geospatial insights☆14Mar 13, 2021Updated 5 years ago