benchmarks for LLM tokenizers
☆21Mar 25, 2026Updated 5 months ago
Alternatives and similar repositories for tokka-bench
Users that are interested in tokka-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 🍡 30x faster tokenization for every HuggingFace model☆51Aug 27, 2026Updated last week
- Using Plotly to create a heatmap visualization of monthly and hourly data☆13Aug 9, 2021Updated 5 years ago
- Supervised and unsupervised self-organising maps☆13Jun 22, 2026Updated 2 months ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 3 years ago
- Crypto Chart Widget for GitHub.☆10Dec 27, 2024Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Tokenization across languages. Useful as preprocessing for subword tokenization.☆19Feb 7, 2023Updated 3 years ago
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆53Aug 31, 2026Updated last week
- Angular feature flag library with Unleash integration, provides reactive programing, type-safe feature flag management with zero-configur…☆33Updated this week
- Calculate parameters that are not available in datasheet of photovoltaic modules using equations derived from diode model☆13Mar 9, 2017Updated 9 years ago
- The CODWOE shared task invites you to compare two types of semantic descriptions: dictionary glosses and word embedding representations. …☆12Jul 13, 2022Updated 4 years ago
- Deep CCA (DCCA) network☆15Apr 12, 2016Updated 10 years ago
- OCF's DataPipe based dataloader for training and inference☆15Jun 2, 2025Updated last year
- ☆10Mar 18, 2021Updated 5 years ago
- A Combinatory Categorial Grammar library.☆22Nov 11, 2013Updated 12 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Glacier: Guided Locally Constrained Counterfactual Explanations for Time Series Classification (Machine Learning journal)☆12Mar 15, 2024Updated 2 years ago
- pyndl implements a Naive discriminative learning which is a learning and classification models based on the Rescorla-Wagner equations in …☆13Dec 8, 2025Updated 8 months ago
- Easy JS performance testing using Benchmark.js☆16Sep 9, 2013Updated 12 years ago
- A tool for making a probabilistic power forecast for a fleet of solar plants using an existing deterministic forecast☆16Dec 2, 2024Updated last year
- ☆18Feb 1, 2023Updated 3 years ago
- Time series analytics with Python☆17Dec 7, 2015Updated 10 years ago
- UMLS in Python with MongoDB.☆18Aug 5, 2026Updated last month
- ☆10Jun 9, 2023Updated 3 years ago
- Tools for working with PubMed data.☆15Sep 26, 2020Updated 5 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Learning BPE embeddings by first learning a segmentation model and then training word2vec☆19Dec 18, 2022Updated 3 years ago
- Turn a Nearley grammar into an efficient fake-text generator☆22Jan 8, 2021Updated 5 years ago
- Basis data, analysis, and visualization☆15Jan 3, 2014Updated 12 years ago
- Learn how to use Redash to query and display your database data☆24Jan 16, 2018Updated 8 years ago
- This repository hosts a ChatGPT plugin that enables the uploading of a PDF URL. The plugin extracts information from the PDF, to enhance …☆10May 24, 2023Updated 3 years ago
- Python package that offers text scrubbing functionality, providing building blocks for string cleaning as well as normalizing geographica…☆22Aug 26, 2024Updated 2 years ago
- Show tweets or whatever you'd like on an E-ink display on your wall☆23Feb 10, 2017Updated 9 years ago
- ☆12Aug 1, 2025Updated last year
- (ICML 2023) Feature learning in deep classifiers through Intermediate Neural Collapse: Accompanying code☆16Jul 27, 2023Updated 3 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- Echo State Network☆17May 2, 2014Updated 12 years ago
- Evolutionary Search for expert-level performance on any task with environmental feedback☆14Oct 12, 2025Updated 10 months ago
- Jupyter notebooks for gaining geospatial insights☆14Mar 13, 2021Updated 5 years ago
- given a bounding box, provides a timeseries animation of way creation☆16Nov 17, 2017Updated 8 years ago
- Unified API platform for free access to enterprise-grade AI models from Google, Groq, and OpenRouter. Industrial-ready integration with h…☆13Mar 14, 2025Updated last year
- AI_Powered_Dev_Search_Engine☆12Mar 10, 2024Updated 2 years ago
- PasteScript template for initializing a new buildout/pytest/travis/setuptools-enabled Python project☆25Mar 24, 2014Updated 12 years ago