Approximation of the Claude 3 tokenizer by inspecting generation stream
☆173Jul 22, 2024Updated 2 years ago
Alternatives and similar repositories for anthropic-tokenizer
Users that are interested in anthropic-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆75Jun 16, 2026Updated 3 months ago
- ☆107Mar 4, 2024Updated 2 years ago
- Mamba support for transformer lens☆20Sep 17, 2024Updated 2 years ago
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Updated this week
- Coord: A Unified Interface for All Models☆18Jun 19, 2026Updated 3 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆19Jul 24, 2025Updated last year
- Make running benchmark simple yet maintainable, again. Now only supports Korean-based cross-encoder.☆36Aug 30, 2026Updated 3 weeks ago
- Improving Steering Vectors by Targeting Sparse Autoencoder Features☆30Nov 20, 2024Updated last year
- A fully trainable state space model (SSM)☆16Mar 18, 2025Updated last year
- A library for mechanistic anomaly detection☆22Jan 9, 2025Updated last year
- Use XML tags for long context prompting using Claude's multi-document structure.☆27Aug 9, 2025Updated last year
- Proteus 2.0☆10Updated this week
- Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA☆42Feb 27, 2024Updated 2 years ago
- ☆13Aug 29, 2023Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- ☆18Feb 18, 2026Updated 7 months ago
- Official repository for the paper "Gradient-based Jailbreak Images for Multimodal Fusion Models" (https//arxiv.org/abs/2410.03489)☆20Oct 22, 2024Updated last year
- [Suspended] Modern, customizable AI character frontend for enthusiasts (inspired by SillyTavern)☆11Nov 8, 2024Updated last year
- ☆30Jan 7, 2026Updated 8 months ago
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- ☆29Oct 6, 2024Updated last year
- Code for SaGe subword tokenizer (EACL 2023)☆28Nov 30, 2024Updated last year
- This library supports evaluating disparities in generated image quality, diversity, and consistency between geographic regions.☆20Jun 3, 2024Updated 2 years ago
- Apps that run on modal.com☆13Sep 14, 2025Updated last year
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Contains random samples referenced in the paper "Sleeper Agents: Training Robustly Deceptive LLMs that Persist Through Safety Training".☆152Mar 9, 2024Updated 2 years ago
- Code used to run the platform for the LLM CTF colocated with SaTML 2024☆29Mar 20, 2024Updated 2 years ago
- Code for "Evidence of Learned Look-Ahead in a Chess-Playing Neural Network"☆31Jun 4, 2024Updated 2 years ago
- ✂️ OpenAI's tiktoken tokenizer written in Go☆20Jan 31, 2025Updated last year
- Make tool-calling schemas for existing tools☆14Mar 8, 2025Updated last year
- Transform unstructured documents into actionable, structured data with enterprise-grade precision and reliability, ready for large-scale …☆22Oct 13, 2025Updated 11 months ago
- ☆13Apr 17, 2024Updated 2 years ago
- The training codes of Jasper-Token-Compression-600M☆22Nov 19, 2025Updated 10 months ago
- A tiny easily hackable implementation of a feature dashboard.☆18Oct 21, 2025Updated 11 months ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Code for the paper "Fishing for Magikarp"☆191Jul 10, 2026Updated 2 months ago
- Code and results accompanying the paper "Refusal in Language Models Is Mediated by a Single Direction".☆447Jun 13, 2025Updated last year
- Official repo for "Binary Retrieval-augmented Reward Mitigates Hallucinations"☆16Nov 13, 2025Updated 10 months ago
- This is the official repository for the "Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP" paper acce…☆25Feb 16, 2026Updated 7 months ago
- ☆17Aug 13, 2026Updated last month
- Auto-generate programs in C derived languages for multiple platforms☆18Sep 9, 2026Updated 2 weeks ago
- ☆176Feb 4, 2023Updated 3 years ago