Claude's Tokenizer, Offline, Kinda
☆102Sep 25, 2026Updated last month
Alternatives and similar repositories for ctok
Users that are interested in ctok are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Updated this week
- A training codebase for Zeroth-Order Optimization☆14Updated this week
- A shared, persistent world for people and AIs — the networking, persistence, and multi-participant layer over the eidoverse-video toolkit…☆48Updated this week
- ☆12Mar 7, 2022Updated 4 years ago
- Minimal implementation of multiple PEFT methods for LLaMA fine-tuning☆13May 7, 2023Updated 3 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Official code for StochasTok: Improving Fine-Grained Subword Understanding in LLMs☆14Mar 28, 2026Updated 7 months ago
- ☆20Sep 11, 2026Updated last month
- ☆12Aug 19, 2026Updated 2 months ago
- ☆27Aug 19, 2026Updated 2 months ago
- Attention variant with per-channel multiplicative decay☆50Jun 3, 2026Updated 5 months ago
- ☆16Aug 26, 2026Updated 2 months ago
- MoE training for Me and You and maybe other people☆400Mar 15, 2026Updated 7 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆94Oct 30, 2024Updated 2 years ago
- Fused MoE kernel☆24Oct 30, 2026Updated last week
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Official implementation of "Data Mixture Inference: What do BPE tokenizers reveal about their training data?"☆22May 15, 2025Updated last year
- Delta Attention Residuals - supplementary code and pretrained models☆45Sep 26, 2026Updated last month
- ☆14Aug 7, 2026Updated 3 months ago
- C++ code of "Learning to Parse and Translate Improves Neural Machine Translation"☆21May 8, 2017Updated 9 years ago
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆112Mar 7, 2025Updated last year
- A high performance development and dispatch library for high-performance machine learning kernels.☆38Aug 15, 2026Updated 2 months ago
- End-to-end formally verified solvers for the ideal magnetohydrodynamics equations (with and without hyperbolic divergence cleaning) in 1D…☆12Aug 27, 2026Updated 2 months ago
- Supporting code for "LLMs for your iPhone: Whole-Tensor 4 Bit Quantization"☆11Mar 31, 2024Updated 2 years ago
- ☆22Oct 26, 2020Updated 6 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Code for EMNLP2021 paper "Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training"☆20Nov 12, 2021Updated 4 years ago
- Example implementation of "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, …☆19Jan 22, 2026Updated 9 months ago
- Official repository for Parallax (Parameterized Local Linear Attention)☆71Jul 30, 2026Updated 3 months ago
- ☆41Updated this week
- A JAX Research Toolkit for Visualizing, Manipulating, and Understanding Gemma Models with Multi-modal Support based on Penzai.☆101Jan 13, 2026Updated 9 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆387Dec 12, 2024Updated last year
- Silly twitter torch implementations.☆48Oct 14, 2022Updated 4 years ago
- ☆11Jan 15, 2026Updated 9 months ago
- ☆31Updated this week
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆22Aug 21, 2026Updated 2 months ago
- The official implementation of HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization☆19Mar 7, 2025Updated last year
- web app for chatting with llms on openrouter with leaked unabridged reasoning☆107Oct 24, 2026Updated 2 weeks ago
- ☆13Oct 19, 2020Updated 6 years ago
- End-to-end formally verified solvers for the electrostatic Vlasov equation in 1x1v, 1x2v, 1x3v, 2x2v, 2x3v, and 3x3v (i.e. 2D to 6D) phas…☆17Jul 21, 2026Updated 3 months ago
- [Tech Report, NeurIPS 2026] Expanded Hyper-Connections☆77Oct 16, 2026Updated 3 weeks ago
- Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible…☆118Updated this week