Alternatives and similar repositories for chinese-tokens-in-tiktoken:

Users that are interested in chinese-tokens-in-tiktoken are comparing it to the libraries listed below

• 2404589803 / hf-daily-paper-newsletter-chinese
  🔥Your Daily Dose of AI Research from Hugging Face 🔥 Stay updated with the latest AI breakthroughs! This bot automatically collects and…
  ☆48Updated this week
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆78Updated 10 months ago
• krystalan / DRT-o1
  DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought
  ☆207Updated last month
• Zheng0428 / COIG-Kun
  ☆36Updated 5 months ago
• Bui1dMySea / MemLong
  ☆91Updated 2 months ago
• open-compass / GAOKAO-Eval
  ☆100Updated 2 months ago
• RapidAI / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆68Updated last year
• FateScript / token_visualizer
  Token level visualization tools for large language models
  ☆74Updated last month
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆36Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆128Updated 8 months ago
• TIGER-AI-Lab / ScholarCopilot
  Scholar Copilot is an intelligent academic writing assistant that enhances the research writing process through AI-powered text completio…
  ☆78Updated 2 months ago
• crazycth / WizardLearner
  Pretrain、decay、SFT a CodeLLM from scratch 🧙‍♂️
  ☆36Updated 9 months ago
• apple / ml-lucid-datagen
  ☆30Updated 11 months ago
• NJUDeepEngine / CAEF
  Code for paper: "Executing Arithmetic: Fine-Tuning Large Language Models as Turing Machines"
  ☆11Updated 4 months ago
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆30Updated 8 months ago
• codefuse-ai / CodeFuse-CGE
  ☆17Updated last month
• zhaochenyang20 / Prompt2Model-Self-Guide
  SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper
  ☆29Updated 8 months ago
• ClosedCharacter / Peach
  我们是第一个完全可商用的角色大模型。
  ☆39Updated 6 months ago
• GAIR-NLP / ReAlign
  Reformatted Alignment
  ☆114Updated 4 months ago
• karminski / streaming-json-py
  A streamlined, user-friendly JSON streaming preprocessor, crafted in Python.
  ☆88Updated 5 months ago
• ninehills / llm-playground
  Prompt 工程师利器，可同时比较多个 Prompts 在多个 LLM 模型上的效果
  ☆95Updated last year
• SUSTech-IDEA / SUS-Chat
  SUS-Chat: Instruction tuning done right
  ☆48Updated last year
• Tomorrowdawn / top_nsigma
  The official code repo and data hub of top_nsigma sampling strategy for LLMs.
  ☆20Updated last week
• AUGMXNT / deccp
  Evaling and unaligning Chinese LLM censorship
  ☆55Updated 4 months ago
• xverse-ai / XVERSE-MoE-A36B
  XVERSE-MoE-A36B: A multilingual large language model developed by XVERSE Technology Inc.
  ☆37Updated 5 months ago
• THUDM / GLM-Edge
  GLM Series Edge Models
  ☆130Updated this week
• SparksJoe / Prism
  A Framework for Decoupling and Assessing the Capabilities of VLMs
  ☆40Updated 7 months ago
• zjrwtx / SFT-data-builder
  利用免费的大模型api来结合你的私域数据来生成sft训练数据（妥妥白嫖）支持llamafactory等工具的训练数据格式synthetic data
  ☆129Updated 2 months ago