Alternatives and similar repositories for chinese-tokens-in-tiktoken

Users that are interested in chinese-tokens-in-tiktoken are comparing it to the libraries listed below

• woct0rdho / transformers-qwen3-moe-fused
  Fused Qwen3 MoE layer for faster training, compatible with HF Transformers, LoRA, 4-bit quant, Unsloth
  ☆122Updated this week
• krystalan / DRT
  Deep Reasoning Translation (DRT) Project
  ☆226Updated last month
• ninehills / llm-playground
  Prompt 工程师利器，可同时比较多个 Prompts 在多个 LLM 模型上的效果
  ☆95Updated last year
• RapidAI / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆69Updated 2 years ago
• FateScript / token_visualizer
  Token level visualization tools for large language models
  ☆83Updated 6 months ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆135Updated last year
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆42Updated last year
• open-compass / GAOKAO-Eval
  ☆112Updated 7 months ago
• apple / ml-lucid-datagen
  ☆32Updated last year
• Tongyi-Zhiwen / QwenLong-L1
  ☆280Updated last month
• BeautyyuYanli / tooluser
  Enable tool-use ability for any LLM model (DeepSeek V3/R1, etc.)
  ☆51Updated last month
• THUDM / GLM-Edge
  GLM Series Edge Models
  ☆144Updated last month
• 2404589803 / hf-daily-paper-newsletter-chinese
  🔥Your Daily Dose of AI Research from Hugging Face 🔥 Stay updated with the latest AI breakthroughs! This bot automatically collects and…
  ☆52Updated this week
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆79Updated last year
• ClosedCharacter / Peach
  我们是第一个完全可商用的角色大模型。
  ☆40Updated 11 months ago
• Bui1dMySea / MemLong
  ☆94Updated 7 months ago
• CrazyBoyM / CodeLLaMA-chat
  CodeLLaMA 中文版 - 代码生成助手，huggingface累积下载2w+次
  ☆45Updated last year
• xverse-ai / XVERSE-MoE-A36B
  XVERSE-MoE-A36B: A multilingual large language model developed by XVERSE Technology Inc.
  ☆39Updated 10 months ago
• crazycth / WizardLearner
  Pretrain、decay、SFT a CodeLLM from scratch 🧙‍♂️
  ☆36Updated last year
• Jellyfish042 / Sudoku-RWKV
  ☆142Updated 7 months ago
• MoonshotAI / moonpalace
  MoonPalace（月宫）是由 Moonshot AI 月之暗面提供的 API 调试工具。
  ☆192Updated 6 months ago
• codefuse-ai / CodeFuse-CGE
  ☆21Updated 2 months ago
• StarRing2022 / R1-Nature
  最简易的R1结果在小模型上的复现，阐述类O1与DeepSeek R1最重要的本质。Think is all your need。利用实验佐证，对于强推理能力，think思考过程性内容是AGI/ASI的核心。
  ☆45Updated 5 months ago
• mazzzystar / TurtleBench
  TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
  ☆152Updated 9 months ago
• sheryc / arxiv-markdown-parser-plugin
  Chrome / Edge extension to turn arXiv papers into Markdown codes in one click.
  ☆78Updated 3 months ago
• IEIT-Yuan / Yuan2.0-M32
  Mixture-of-Experts (MoE) Language Model
  ☆189Updated 10 months ago
• xlang-ai / computer-agent-arena
  Computer Agent Arena: Test & compare AI agents in real desktop apps & web environments. Code/data coming soon!
  ☆45Updated 3 months ago
• pandalla / DataTager
  Fine-Tune LLM Synthetic-Data application and "From Data to AGI: Unlocking the Secrets of Large Language Model"
  ☆17Updated last year
• bigai-nlco / TokenSwift
  [ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation
  ☆110Updated last month
• Azure99 / BlossomData
  A fluent, scalable, and easy-to-use LLM data processing framework.
  ☆22Updated last week