A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.
☆12Oct 16, 2026Updated 2 weeks ago
Alternatives and similar repositories for Tokenicer
Users that are interested in Tokenicer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Evalution: evolve your LLMs with better evals.☆17Updated this week
- Fully chunked KL-Loss kernel, that massively reduces VRAM usage☆20Aug 10, 2026Updated 2 months ago
- A Simple library for communicating with USB and Bluetooth HID devices on Linux, Mac, and Windows.☆13Jun 9, 2025Updated last year
- Remembra Relay: one agent stops, the next one already knows. Session handoffs built from git facts, for the next coding agent in any MCP …☆16Oct 25, 2026Updated last week
- Super simple python connectors for llama.cpp, including vision models (Gemma 3, Qwen2-VL). Compile llama.cpp and run!☆31Dec 11, 2025Updated 10 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- INT8-native vLLM fork☆24Oct 21, 2026Updated last week
- Out-of-tree Linux thunderbolt_net driver (DKMS) tracking upstream, with a receive-side workaround for macOS Thunderbolt Bridge TCP aggreg…☆18Oct 18, 2026Updated 2 weeks ago
- Compute Where It Counts: Scaling both speed and quality with training compute☆13Sep 18, 2025Updated last year
- local distributed learning and inference☆14May 27, 2026Updated 5 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆38Updated this week
- OCEAN – Open-source CXL Emulation at Hyperscale Architecture and Networking☆14Feb 9, 2026Updated 8 months ago
- Learnable Semi-structured Sparsity for Vision Transformers and Diffusion Transformers☆15Feb 7, 2025Updated last year
- Compact image interpolation model for generating in-between frames, with ComfyUI support and the same core model used in FrameFusion Mo…☆27Apr 17, 2026Updated 6 months ago
- Python library to compress LitGPT models for resource efficient inference.☆16Oct 2, 2026Updated last month
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- MNIST in the browser☆20Apr 27, 2020Updated 6 years ago
- Code for Fast as CHITA: Neural Network Pruning with Combinatorial Optimization☆14Aug 2, 2023Updated 3 years ago
- ☆55Feb 29, 2024Updated 2 years ago
- Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark. Paper at: https://arxiv.org/abs/2503.20786☆11Mar 27, 2025Updated last year
- Thin wrapper around GGML to make life easier☆48Jul 26, 2026Updated 3 months ago
- WolframAlpha inside Unity☆13Nov 15, 2021Updated 4 years ago
- OpenWrt Multi WAN Port Policy Routing Tool（OpenWrt固件多WAN口策略路由分流工具）☆12Jul 26, 2024Updated 2 years ago
- ☆17Jun 16, 2026Updated 4 months ago
- ⛔ DEPRECATED -- use flash-head instead (pip install flash-head)☆29Apr 10, 2026Updated 6 months ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Android plugin for Processing.org which provides file and folder selection on mobile devices.☆10Jun 12, 2018Updated 8 years ago
- Mechanical Squid Factory XInput PC Fight Stick☆31Apr 12, 2016Updated 10 years ago
- Voronoi Tessellations of portraits☆13May 9, 2012Updated 14 years ago
- Welcome to Uncertainty Metrics! The goal of this library is to provide an easy-to-use interface for both measuring uncertainty across Goo…☆24Nov 18, 2020Updated 5 years ago
- Testing LLM reasoning abilities with lineage relationship quizzes.☆46Mar 10, 2026Updated 7 months ago
- Easily write scripts with Go. Improvements for https://github.com/bitfield/script.☆14Jul 15, 2023Updated 3 years ago
- The DPAB-α Benchmark☆32Jan 15, 2025Updated last year
- Plugin for Cordova that enables fully native-looking apps☆18Feb 26, 2015Updated 11 years ago
- Alternative Logic16 Firmware☆36Aug 14, 2015Updated 11 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Code from today's Live Stream☆11Jan 2, 2022Updated 4 years ago
- Documents and guides that might be useful for People working at Universities in connection with Open Source☆11Jun 14, 2021Updated 5 years ago
- Cookiecutter uv Hypermodern Python☆26Jul 30, 2026Updated 3 months ago
- Experimental mode to support Kotlin in Processing☆10Jun 8, 2022Updated 4 years ago
- Dotted notation with pattern matching☆14Updated this week
- Get an in-depth overview of the most commonly needed functionality of the Unity Object Pooling API introduced in the 2021 version of the…☆11Oct 26, 2021Updated 5 years ago
- Self-contained Python lib with zero-dependencies that give you a unified device properties for gpu, cpu, and npu. No more calling separat…☆18Oct 18, 2026Updated 2 weeks ago