Alternatives and similar repositories for tiktokenizer

Users that are interested in tiktokenizer are comparing it to the libraries listed below

• LiveBench / LiveBench
  LiveBench: A Challenging, Contamination-Free LLM Benchmark
  ☆1,032Updated this week
• OpenRouterTeam / openrouter-runner
  Deprecated inference engine
  ☆1,185Updated 5 months ago
• philschmid / clipper.js
  HTML to Markdown converter and crawler.
  ☆609Updated 2 years ago
• niieani / gpt-tokenizer
  The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken wit…
  ☆729Updated last month
• domeccleston / sharegpt
  Easily share permanent links to ChatGPT conversations with your friends
  ☆1,759Updated last year
• mlc-ai / web-llm-chat
  Chat with AI large language models running natively in your browser. Enjoy private, server-free, seamless AI conversations.
  ☆941Updated 5 months ago
• e2b-dev / code-interpreter
  Python & JS/TS SDK for running AI-generated code/code interpreting in your AI app
  ☆2,193Updated this week
• 0xdevalias / chatgpt-source-watch
  Analyzing the evolution of ChatGPT's codebase through time with curated archives and scripts
  ☆296Updated 9 months ago
• upstash / wikipedia-semantic-search
  Semantic Search on Wikipedia with Upstash Vector
  ☆473Updated last month
• nomic-ai / nomic
  Nomic Developer API SDK
  ☆1,865Updated 2 months ago
• tavily-ai / tavily-python
  The Tavily Python wrapper allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, an…
  ☆991Updated last week
• Nutlope / turboseek
  An AI search engine inspired by Perplexity
  ☆1,609Updated last month
• AnswerDotAI / llms-txt
  The /llms.txt file, helping language models use your website
  ☆2,118Updated last week
• openai / openai-structured-outputs-samples
  Sample apps to help developers get started with Structured Outputs
  ☆667Updated last month
• hundredblocks / transcription_demo
  ☆264Updated last year
• mlfoundations / dclm
  DataComp for Language Models
  ☆1,413Updated 5 months ago
• adithya-s-k / marker-api
  Easily deployable 🚀 API to convert PDF to markdown quickly with high accuracy.
  ☆936Updated last year
• weavel-ai / Ape
  Your first AI prompt engineer
  ☆411Updated 7 months ago
• latitudegames / GPT-3-Encoder
  Javascript BPE Encoder Decoder for GPT-2 / GPT-3
  ☆723Updated 2 years ago
• JonathanChavezTamales / llm-leaderboard
  A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)
  ☆359Updated 3 months ago
• thakkarparth007 / copilot-explorer
  Hacky repo to see what the Copilot extension sends to the server
  ☆708Updated 2 years ago
• run-llama / LlamaIndexTS
  Data framework for your LLM applications. Focus on server side solution
  ☆3,036Updated last week
• NousResearch / Hermes-Function-Calling
  ☆1,193Updated last month
• mistralai / mistral-common
  Official inference library for pre-processing of Mistral models
  ☆849Updated last week
• modal-labs / modal-examples
  Examples of programs built using Modal
  ☆1,099Updated this week
• anysphere / priompt
  Prompt design using JSX.
  ☆2,761Updated 3 months ago
• gkamradt / LLMTest_NeedleInAHaystack
  Doing simple retrieval from LLM models at various context lengths to measure accuracy
  ☆2,167Updated last year
• mckaywrigley / clarity-ai
  A Perplexity clone.
  ☆1,409Updated last year
• run-llama / chat-llamaindex
  ☆978Updated last month
• attentionmech / mav
  Model Activity Visualiser
  ☆521Updated 10 months ago