Alternatives and similar repositories for chinese-tokens-in-tiktoken:

Users that are interested in chinese-tokens-in-tiktoken are comparing it to the libraries listed below

• ninehills / langeval
  Evaluation for AI apps and agent
  ☆40Updated last year
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆79Updated last year
• ClosedCharacter / Peach
  我们是第一个完全可商用的角色大模型。
  ☆39Updated 8 months ago
• RapidAI / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆69Updated last year
• 2404589803 / hf-daily-paper-newsletter-chinese
  🔥Your Daily Dose of AI Research from Hugging Face 🔥 Stay updated with the latest AI breakthroughs! This bot automatically collects and…
  ☆50Updated this week
• tpoisonooo / kimi-chat-testdata
  kimi-chat 测试数据
  ☆7Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆132Updated 10 months ago
• StarRing2022 / R1-Nature
  最简易的R1结果在小模型上的复现，阐述类O1与DeepSeek R1最重要的本质。Think is all your need。利用实验佐证，对于强推理能力，think思考过程性内容是AGI/ASI的核心。
  ☆44Updated 2 months ago
• SUSTech-IDEA / SUS-Chat
  SUS-Chat: Instruction tuning done right
  ☆48Updated last year
• open-compass / GAOKAO-Eval
  ☆105Updated 4 months ago
• crazycth / WizardLearner
  Pretrain、decay、SFT a CodeLLM from scratch 🧙‍♂️
  ☆35Updated 11 months ago
• FateScript / token_visualizer
  Token level visualization tools for large language models
  ☆79Updated 3 months ago
• NJUDeepEngine / CAEF
  Code for paper: "Executing Arithmetic: Fine-Tuning Large Language Models as Turing Machines"
  ☆11Updated 6 months ago
• CyberCommy / baidu-qa-100w
  百度QA100万数据集
  ☆47Updated last year
• thunlp / APB
  ☆27Updated 2 months ago
• zxx000728 / CodeGPT
  CodeGPT: A Code-Related Dialogue Dataset Generated by GPT and for GPT
  ☆113Updated last year
• ExpressAI / AI-Gaokao
  Gaokao Benchmark for AI
  ☆108Updated 2 years ago
• HIT-SCIR / Abacus
  珠算代码大模型（Abacus Code LLM）
  ☆56Updated 7 months ago
• krystalan / DRT
  Deep Reasoning Translation via Reinforcement Learning (arXiv preprint 2025); DRT: Deep Reasoning Translation via Long Chain-of-Thought (a…
  ☆215Updated last week
• WangRongsheng / o1-imitator
  ☆53Updated last month
• xverse-ai / XVERSE-65B
  XVERSE-65B: A multilingual large language model developed by XVERSE Technology Inc.
  ☆139Updated last year
• THUDM / GLM-Edge
  GLM Series Edge Models
  ☆136Updated 2 months ago
• ninehills / llm-playground
  Prompt 工程师利器，可同时比较多个 Prompts 在多个 LLM 模型上的效果
  ☆95Updated last year
• K024 / chatglm-q
  Another ChatGLM2 implementation for GPTQ quantization
  ☆54Updated last year
• xverse-ai / XVERSE-MoE-A36B
  XVERSE-MoE-A36B: A multilingual large language model developed by XVERSE Technology Inc.
  ☆36Updated 7 months ago
• mazzzystar / TurtleBench
  TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
  ☆145Updated 6 months ago
• codefuse-ai / CodeFuse-MFT-VLM
  ☆36Updated 6 months ago
• wangyuxinwhy / lmclient
  Python client designed specifically for large-scale requests to the openai interface
  ☆21Updated last year
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆37Updated last year
• MikeGu721 / AgentGroup
  ☆91Updated last year