Alternatives and similar repositories for bpe-tokenizer:

Users that are interested in bpe-tokenizer are comparing it to the libraries listed below

• RethinkFun / trian_ppo
  ☆50Updated 5 months ago
• owenliang / qwen-dpo
  通义千问的DPO训练
  ☆40Updated 6 months ago
• lansinuote / Simple_RLHF
  ☆73Updated 4 months ago
• owenliang / chatgpt
  simple decoder-only GTP model in pytorch
  ☆37Updated 10 months ago
• LDLINGLINGLING / adan_application
  一些大语言模型和多模态模型的应用,主要包括Rag，小模型，Agent，跨模态搜索，OCR等等
  ☆157Updated 4 months ago
• AI-Study-Han / Zero-Qwen-VL
  训练一个对中文支持更好的LLaVA模型，并开源训练代码和数据。
  ☆53Updated 6 months ago
• owenliang / another-pytorch
  A simple deep learning framework inspired by Dezero and PyTorch
  ☆30Updated last month
• yuanzhoulvpi2017 / SentenceEmbedding
  ☆105Updated 8 months ago
• xinyanghuang7 / Basic-Visual-Language-Model
  Build a simple basic multimodal large model from scratch. 从零搭建一个简单的基础多模态大模型🤖
  ☆34Updated 9 months ago
• lansinuote / Simple_LLM_PPO
  ☆40Updated 7 months ago
• yang19527 / AwesomeInterview
  包含程序员面试大厂面试题和面试经验
  ☆123Updated 2 months ago
• sunkx109 / llama
  Inference code for LLaMA models
  ☆118Updated last year
• OvJat / DeepSpeedTutorial
  DeepSpeed Tutorial
  ☆95Updated 7 months ago
• wdndev / tiny-rag
  一个很小很小的RAG系统
  ☆177Updated 3 months ago
• liujunwen23 / MIRE
  WWW2025 Multimodal Intent Recognition for Dialogue Systems Challenge
  ☆118Updated 4 months ago
• Logistic98 / rag-omni
  基于大语言模型的检索增强生成RAG示例
  ☆134Updated 3 months ago
• RethinkFun / sft
  ☆29Updated 7 months ago
• cwxndl / LLM
  大语言模型应用：RAG、NL2SQL、聊天机器人、预训练、MOE混合专家模型、微调训练、强化学习、天池数据竞赛
  ☆58Updated last month
• bobo0810 / LearnDeepSpeed
  DeepSpeed教程 & 示例注释 & 学习笔记 （大模型高效训练）
  ☆151Updated last year
• firechecking / CleanTransformer
  an implementation of transformer, bert, gpt, and diffusion models for learning purposes
  ☆151Updated 5 months ago
• SmartFlowAI / LLM101n-CN
  LLM101n: Let's build a Storyteller 中文版
  ☆130Updated 7 months ago
• heyblackC / BetterMixture-Top1-Solution
  天池算法比赛《BetterMixture - 大模型数据混合挑战赛》的第一名top1解决方案
  ☆27Updated 8 months ago
• Mxoder / LLM-from-scratch
  一些 LLM 方面的从零复现笔记
  ☆175Updated 6 months ago
• Tlntin / qwen-ascend-llm
  ☆39Updated 4 months ago
• lansinuote / Simple_LLM_DPO
  ☆66Updated last year
• OctopusMind / RLHF_PPO
  ppo算法实现
  ☆23Updated 9 months ago
• KMnO4-zx / xlab-huanhuan
  ☆60Updated last year
• yanqiangmiffy / how-to-train-tokenizer
  怎么训练一个LLM分词器
  ☆142Updated last year
• ssbuild / qwen_finetuning
  qwen models finetuning
  ☆93Updated 2 weeks ago