benchmarks for LLM tokenizers
☆22Mar 25, 2026Updated 7 months ago
Alternatives and similar repositories for tokka-bench
Users that are interested in tokka-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A TypeScript re-implementation of React: fibers, lanes, scheduling, hooks, streaming SSR, and selective hydration — without the legacy cr…☆35Updated this week
- 🍡 30x faster tokenization for every HuggingFace model☆56Updated this week
- Train small sequence models in your browser with WebGPU.☆37Dec 3, 2025Updated 11 months ago
- Automatically exported from code.google.com/p/transducersaurus☆11Apr 1, 2015Updated 11 years ago
- LLaMA 2-based chat for ultra-low-memory hardware with a 60 MB footprint☆15Nov 12, 2025Updated 11 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Benchmarking framework for multiple templating engines☆14Aug 21, 2020Updated 6 years ago
- Nanoloop source files for the album "Prime 16"☆12Mar 7, 2026Updated 8 months ago
- ☆12Jun 27, 2024Updated 2 years ago
- ☆11Sep 4, 2026Updated 2 months ago
- ☆11Aug 1, 2019Updated 7 years ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 3 years ago
- IDE for Rust language☆20Oct 24, 2026Updated last week
- Crypto Chart Widget for GitHub.☆10Dec 27, 2024Updated last year
- Self-supervised MPFNet for realistic bokeh effect rendering(JVCIR2022)☆14Jul 5, 2022Updated 4 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Disambiguating biomedical and clinical concepts with word embeddings☆15Aug 5, 2026Updated 3 months ago
- a library for fast static text embedding inference in Go☆24Sep 18, 2026Updated last month
- Floating DevTools panel that intercepts OpenAI, Anthropic, Gemini, Mistral, Groq, and Cohere calls in your Angular app and shows you the …☆225Jun 19, 2026Updated 4 months ago
- Angular feature flag library with Unleash integration, provides reactive programing, type-safe feature flag management with zero-configur…☆33Oct 29, 2026Updated last week
- Unsupervised concept extraction from clinical text☆14Jun 17, 2024Updated 2 years ago
- Sequencable robot operated digital tape machine☆19May 14, 2024Updated 2 years ago
- Generates ES modules from CSS☆20Updated this week
- TailNG UI is a signal-first Angular component library with Material-like components, framework-agnostic styling, and support for Angular …☆15Oct 30, 2026Updated last week
- ☆25Aug 2, 2022Updated 4 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Chrome extension which lets you watch WWDC Developer Videos in Google Chrome (thus not having to use Safari)☆17Oct 11, 2017Updated 9 years ago
- Tokenization across languages. Useful as preprocessing for subword tokenization.☆19Feb 7, 2023Updated 3 years ago
- ☆19May 10, 2026Updated 5 months ago
- Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"☆15Sep 2, 2026Updated 2 months ago
- pyndl implements a Naive discriminative learning which is a learning and classification models based on the Rescorla-Wagner equations in …☆13Dec 8, 2025Updated 10 months ago
- Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and transformer models. Header-only and zero-dependency.☆16Aug 7, 2025Updated last year
- ☆23Sep 11, 2026Updated last month
- Lab notebooks for CS474, Fall 2019☆14Jan 22, 2025Updated last year
- A Combinatory Categorial Grammar library.☆22Nov 11, 2013Updated 12 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- A status page for community projects☆23Sep 26, 2026Updated last month
- ☆21Mar 28, 2022Updated 4 years ago
- A VSCode extension for the Eta templating language☆15Jul 18, 2023Updated 3 years ago
- Frontend Form Solution;Valibot define to form component☆11Updated this week
- Show tweets or whatever you'd like on an E-ink display on your wall☆23Feb 10, 2017Updated 9 years ago
- ☆18Feb 1, 2023Updated 3 years ago
- Acoustic distance measure for comparing pronunciations☆17Aug 2, 2022Updated 4 years ago