Approximation of the Claude 3 tokenizer by inspecting generation stream
☆173Jul 22, 2024Updated 2 years ago
Alternatives and similar repositories for anthropic-tokenizer
Users that are interested in anthropic-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆74Jun 16, 2026Updated 2 months ago
- ☆19Feb 24, 2023Updated 3 years ago
- Official implementation of "Interpreting and Controlling Vision Foundation Models via Text Explanations"☆14May 29, 2024Updated 2 years ago
- Mamba support for transformer lens☆20Sep 17, 2024Updated last year
- ☆54Jun 19, 2024Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ☆26Feb 11, 2025Updated last year
- Coord: A Unified Interface for All Models☆18Jun 19, 2026Updated last month
- ☆30Jun 19, 2023Updated 3 years ago
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆19Jul 24, 2025Updated last year
- [ICML 24] A novel automated neuron explanation framework that can accurately describe poly-semantic concepts in deep neural networks☆14May 2, 2025Updated last year
- CLI that queries multiple language models in parallel using prompts from a CSV file☆28Sep 24, 2025Updated 10 months ago
- Make running benchmark simple yet maintainable, again. Now only supports Korean-based cross-encoder.☆35Updated this week
- Training code for Sparse Autoencoders on Embedding models☆40Jul 11, 2026Updated last month
- Improving Steering Vectors by Targeting Sparse Autoencoder Features☆29Nov 20, 2024Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Use XML tags for long context prompting using Claude's multi-document structure.☆27Aug 9, 2025Updated last year
- Proteus 2.0☆10Updated this week
- utilities for loading and running text embeddings with onnx☆46Aug 16, 2025Updated last year
- Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA☆42Feb 27, 2024Updated 2 years ago
- ☆18Feb 18, 2026Updated 6 months ago
- Official repository for the paper "Gradient-based Jailbreak Images for Multimodal Fusion Models" (https//arxiv.org/abs/2410.03489)☆20Oct 22, 2024Updated last year
- Fluent student-teacher redteaming☆23Jul 25, 2024Updated 2 years ago
- ☆25Jan 7, 2026Updated 7 months ago
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- A dictionary, but it shows you position in embedding space relative to some synonyms/antonyms instead of a definition.☆74Nov 25, 2025Updated 8 months ago
- ☆28Oct 6, 2024Updated last year
- ☆66Sep 13, 2025Updated 11 months ago
- Code for SaGe subword tokenizer (EACL 2023)☆28Nov 30, 2024Updated last year
- This library supports evaluating disparities in generated image quality, diversity, and consistency between geographic regions.☆20Jun 3, 2024Updated 2 years ago
- Apps that run on modal.com☆13Sep 14, 2025Updated 11 months ago
- Contains random samples referenced in the paper "Sleeper Agents: Training Robustly Deceptive LLMs that Persist Through Safety Training".☆150Mar 9, 2024Updated 2 years ago
- Code used to run the platform for the LLM CTF colocated with SaTML 2024☆29Mar 20, 2024Updated 2 years ago
- Code for "Evidence of Learned Look-Ahead in a Chess-Playing Neural Network"☆31Jun 4, 2024Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ✂️ OpenAI's tiktoken tokenizer written in Go☆20Jan 31, 2025Updated last year
- Make tool-calling schemas for existing tools☆14Mar 8, 2025Updated last year
- Transform unstructured documents into actionable, structured data with enterprise-grade precision and reliability, ready for large-scale …☆20Oct 13, 2025Updated 10 months ago
- ☆13Apr 17, 2024Updated 2 years ago
- The training codes of Jasper-Token-Compression-600M☆20Nov 19, 2025Updated 9 months ago
- A tiny easily hackable implementation of a feature dashboard.☆17Oct 21, 2025Updated 9 months ago
- Code for the paper "Fishing for Magikarp"☆191Jul 10, 2026Updated last month