Alternatives and similar repositories for anthropic-tokenizer

Users that are interested in anthropic-tokenizer are comparing it to the libraries listed below

• enjalot / latent-data-modal

  View on GitHub
  Using modal.com to process FineWeb-edu data
  ☆20Apr 5, 2025Updated 10 months ago
• lemon-mint / coord

  View on GitHub
  Coord: A Unified Interface for All Models
  ☆18Feb 2, 2026Updated 2 weeks ago
• replicate / go

  View on GitHub
  Repository for go shared libraries (for now).
  ☆11Dec 1, 2025Updated 2 months ago
• taylorai / onnx_embedding_models

  View on GitHub
  utilities for loading and running text embeddings with onnx
  ☆45Aug 16, 2025Updated 5 months ago
• StableFluffy / EasyLLMFeaturePorter

  View on GitHub
  1-Click is all you need.
  ☆63Apr 29, 2024Updated last year
• SeungyounShin / minimal-r1

  View on GitHub
  ☆26Feb 11, 2025Updated last year
• indentcorp / bizmsg-devtools

  View on GitHub
  비즈엠 개발 서버에서 전화번호 인증을 쉽게 할 수 있는 웹사이트입니다.
  ☆10Feb 27, 2023Updated 2 years ago
• instructkr / reranker-simple-benchmark

  View on GitHub
  Make running benchmark simple yet maintainable, again. Now only supports Korean-based cross-encoder.
  ☆29Dec 2, 2025Updated 2 months ago
• pigmalion-studio / YoursAI

  View on GitHub
  [Suspended] Modern, customizable AI character frontend for enthusiasts (inspired by SillyTavern)
  ☆10Nov 8, 2024Updated last year
• hotchpotch / yasem

  View on GitHub
  YASEM - Yet Another Splade|Sparse Embedder - A simple and efficient library for SPLADE embeddings
  ☆13May 22, 2025Updated 8 months ago
• davidkim205 / translation

  View on GitHub
  ☆12Apr 17, 2024Updated last year
• davidkim205 / kollm_evaluation

  View on GitHub
  자체 구축한 한국어 평가 데이터셋을 이용한 한국어 모델 평가
  ☆31May 31, 2024Updated last year
• mobarski / aidapter

  View on GitHub
  Adapter / facade for language models (OpenAI, Anthropic, Cohere, local transformers, etc)
  ☆20Sep 21, 2023Updated 2 years ago
• DefenderOfBasic / works-in-progress

  View on GitHub
  ☆17Jun 23, 2025Updated 7 months ago
• enjalot / latent-sae

  View on GitHub
  Training code for Sparse Autoencoders on Embedding models
  ☆39Feb 27, 2025Updated 11 months ago
• mattneary / colorspace

  View on GitHub
  assign color hues to a collection of text fragments based on embeddings
  ☆20Jun 15, 2024Updated last year
• evanatyourservice / llm-jax

  View on GitHub
  Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.
  ☆18Jul 24, 2025Updated 6 months ago
• BruceDLong / CodeDog

  View on GitHub
  Auto-generate programs in C derived languages for multiple platforms
  ☆18Aug 4, 2025Updated 6 months ago
• facebookresearch / mexma

  View on GitHub
  MEXMA: Token-level objectives improve sentence representations
  ☆42Jan 6, 2025Updated last year
• tech-beeline / varp

  View on GitHub
  VSCode extension for working with Architecture As A Code in the C4 model. Includes syntax highlighting, diagram preview, and tools for wo…
  ☆31Feb 6, 2026Updated last week
• do-me / flatgeobuf-vectordb

  View on GitHub
  A single static file as vector database, using the cloud-native flatgeobuf file format and http range requests
  ☆17Oct 28, 2025Updated 3 months ago
• pers0na2dev / volta

  View on GitHub
  ⚡Library for easy interaction with RabbitMQ 🐰
  ☆19Oct 13, 2023Updated 2 years ago
• cohere-ai / magikarp

  View on GitHub
  Code for the paper "Fishing for Magikarp"
  ☆180May 15, 2025Updated 9 months ago
• facebookresearch / DIG-In

  View on GitHub
  This library supports evaluating disparities in generated image quality, diversity, and consistency between geographic regions.
  ☆20Jun 3, 2024Updated last year
• microsoft / mutransformers

  View on GitHub
  some common Huggingface transformers in maximal update parametrization (µP)
  ☆87Mar 14, 2022Updated 3 years ago
• wdhorton / formfill

  View on GitHub
  FormFill is a CLI tool that uses LLMs to automatically fill out PDF forms.
  ☆29Nov 22, 2024Updated last year
• AlmogBaku / pytest-evals

  View on GitHub
  A pytest plugin for running and analyzing LLM evaluation tests.
  ☆153Feb 5, 2025Updated last year
• jwj7140 / Gugugo

  View on GitHub
  Gugugo: 한국어 오픈소스 번역 모델 프로젝트
  ☆85Apr 7, 2024Updated last year
• cohere-ai / BinaryVectorDB

  View on GitHub
  Efficient vector database for hundred millions of embeddings.
  ☆211May 17, 2024Updated last year
• stefan-it / modern-bert-ner

  View on GitHub
  My NER Experiments with ModernBERT and Ettin
  ☆26Jul 17, 2025Updated 6 months ago
• Siddhartha80 / AI-Powered-Predictive-Maintenance-System-for-Vehicles-with-Real-Time-Data-Visualization-and-Analysis

  View on GitHub
  Gradient Boosting Models on Real-Time Sensor Data for AI-Enhanced Vehicle Predictive Maintenance. By using a web-based interface to forec…
  ☆19Nov 17, 2024Updated last year
• camel-ai / camel_chat

  View on GitHub
  💬 Minimalistic repository to reproduce and serve CAMEL models.
  ☆25Jun 26, 2023Updated 2 years ago
• FreedomIntelligence / GPT-API-Accelerate

  View on GitHub
  The "GPT-API-Accelerate" project provides a set of Python classes for accelerating the process of generating responses to prompts using t…
  ☆23Oct 12, 2024Updated last year
• ssisOneTeam / Korean-Embedding-Model-Performance-Benchmark-for-Retriever

  View on GitHub
  Korean Sentence Embedding Model Performance Benchmark for RAG
  ☆50Jan 27, 2025Updated last year
• superfly / bfaas

  View on GitHub
  Bash functions-as-a-service
  ☆31Dec 20, 2024Updated last year
• MeLeLBGU / SaGe

  View on GitHub
  Code for SaGe subword tokenizer (EACL 2023)
  ☆27Nov 30, 2024Updated last year
• sionic-ai / flasma

  View on GitHub
  High-performance vector search engine with no loss of accuracy through GPU and dynamic placement
  ☆31Jul 12, 2025Updated 7 months ago
• cr0sh / grasshopper-public

  View on GitHub
  Rust-based Lua runtime for cryptocurrency trading
  ☆22Jun 14, 2025Updated 8 months ago
• google-deepmind / xtr

  View on GitHub
  XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
  ☆60Jun 20, 2024Updated last year