owenliang/bpe-tokenizer external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

owenliang/bpe-tokenizer

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/owenliang/bpe-tokenizer)

Close

owenliang / bpe-tokenizerView on GitHubMore

• External links
• Readme badge

LLM Tokenizer with BPE algorithm

☆50May 7, 2024Updated 2 years ago

Alternatives and similar repositories for bpe-tokenizer

Users that are interested in bpe-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• owenliang / chatgpt

  View on GitHub
  simple decoder-only GTP model in pytorch
  ☆46May 19, 2024Updated 2 years ago
• owenliang / nano-graphrag

  View on GitHub
  A simple, easy-to-hack GraphRAG implementation
  ☆15Sep 21, 2024Updated 2 years ago
• owenliang / another-pytorch

  View on GitHub
  A simple deep learning framework inspired by Dezero and PyTorch
  ☆33Jan 27, 2025Updated last year
• owenliang / mnist-onnx-runtime

  View on GitHub
  MoE model with onnx runtime
  ☆65May 5, 2024Updated 2 years ago
• DezhiKong00 / Sentencepiece-chinese-bbpe

  View on GitHub
  使用Sentencepiece对中文语料进行分词
  ☆13Nov 30, 2023Updated 2 years ago
• 1-Click AI Models by DigitalOcean Gradient • Ad
  Deploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
• CliffKai / assignment1-basics

  View on GitHub
  Student version of Assignment 1 for Stanford CS336 - Language Modeling From Scratch
  ☆24Mar 28, 2026Updated 6 months ago
• owenliang / qwen-dpo

  View on GitHub
  通义千问的DPO训练
  ☆67Sep 21, 2024Updated 2 years ago
• lansinuote / Simple_TRL

  View on GitHub
  ☆18Aug 9, 2024Updated 2 years ago
• yuanzhoulvpi2017 / SentenceEmbedding

  View on GitHub
  ☆121Jun 30, 2024Updated 2 years ago
• chunhuizhang / llms_tuning

  View on GitHub
  stay tuned.
  ☆18Jul 7, 2025Updated last year
• Phoenix8215 / build_neural_network_from_scratch_CPP

  View on GitHub
  Created a simple neural network using C++17 standard and the Eigen library that supports both forward and backward propagation.
  ☆11Jul 27, 2024Updated 2 years ago
• ramsrigouthamg / BERT_generate_grammar_MCQ_from_news_article

  View on GitHub
  Use pretrained BERT model to automatically generate grammar multiple choice questions (MCQ) from any news article or story.
  ☆13Oct 2, 2019Updated 7 years ago
• imxtx / llm-tutorials

  View on GitHub
  ☆29Jan 13, 2024Updated 2 years ago
• GioGioBond / NBCEonChatGLM6b

  View on GitHub
  (NBCE)Naive Bayes-based Context Extension on ChatGLM-6b
  ☆15Jun 7, 2023Updated 3 years ago
• Serverless GPU API endpoints on Runpod - Get Bonus Credits • Ad
  Skip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
• kyocen / Graduation-Design-VQA-based-on-deep-learning

  View on GitHub
  毕业设计: 基于深度学习的视觉问答
  ☆13Jun 20, 2018Updated 8 years ago
• lansinuote / Simple_LLM_DPO

  View on GitHub
  ☆75Nov 13, 2023Updated 2 years ago
• Paranioar / DBL

  View on GitHub
  [TIP2024] The code of “Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching”
  ☆13Jun 12, 2024Updated 2 years ago
• autumn9999 / MTC-with-Category-Shifts

  View on GitHub
  ☆12Oct 5, 2022Updated 4 years ago
• zejunwang1 / easytokenizer

  View on GitHub
  高性能文本 Tokenizer 库
  ☆32Feb 2, 2024Updated 2 years ago
• tianchiguaixia / ocr_recognition

  View on GitHub
  微调阿里开源的文字检测模型，利用合合识别返回的OCR结果作为初始训练数据，对模型进行优化训练，使其更加适应1万张图片的具体场景，提高文字识别的精度。
  ☆10Dec 9, 2024Updated last year
• icantnamemyself / FormerTime

  View on GitHub
  ☆10Feb 21, 2023Updated 3 years ago
• kyegomez / FastFF

  View on GitHub
  Zeta implementation of a reusable and plug in and play feedforward from the paper "Exponentially Faster Language Modeling"
  ☆16Nov 11, 2024Updated last year
• YanzheZhang / SHPReaderJS

  View on GitHub
  This project is a demo of shp.js (https://github.com/kig/shp.js). We read a shp file and draw it on canvas.
  ☆13Jun 14, 2019Updated 7 years ago
• Virtual machines for every use case on DigitalOcean • Ad
  Get dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
• Megum1 / UNIT

  View on GitHub
  [ECCV'24] UNIT: Backdoor Mitigation via Automated Neural Distribution Tightening
  ☆10Dec 18, 2025Updated 10 months ago
• chenzen94 / debug-deepspeed-chat

  View on GitHub
  Debug DeepSpeed-Chat step by step in IDE (在IDE里一步一步调试DeepSpeed-Chat)
  ☆10Apr 17, 2023Updated 3 years ago
• owenliang / rag-retrieval

  View on GitHub
  RAG向量召回示例
  ☆154Feb 14, 2024Updated 2 years ago
• sxysxy / Taurix

  View on GitHub
  Taurix OS kernel. Taurix 系统内核，操作系统原理实(xjb)践(写)
  ☆12Dec 20, 2020Updated 5 years ago
• caihao20 / zh_correct_pinyin

  View on GitHub
  中文纠错-使用拼音树及编辑距离
  ☆13Jul 19, 2019Updated 7 years ago
• wencheng256 / DaCCN

  View on GitHub
  ☆11Aug 10, 2023Updated 3 years ago
• zigangzhao-ai / highBuildingThrowDet

  View on GitHub
  high Building Throw Det
  ☆12Feb 26, 2023Updated 3 years ago
• ZejunCao / bilibili_code

  View on GitHub
  bilibili视频讲解所使用的课件代码记录
  ☆55Oct 3, 2026Updated 2 weeks ago
• owenliang / DeepSeek-Distill-Qwen-For-Child

  View on GitHub
  ☆64Mar 8, 2025Updated last year
• GPUs on demand by Runpod - Special Offer Available • Ad
  Run AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
• lansinuote / Simple_LLM_PPO

  View on GitHub
  ☆45Aug 9, 2024Updated 2 years ago
• PRIS-CV / RelMatch

  View on GitHub
  Code release for "Clue Me In: Semi-Supervised FGVC with Out-of-Distribution Data".
  ☆13Apr 11, 2022Updated 4 years ago
• Littlefean / TimeManager

  View on GitHub
  一个时间管理类app项目，该app能够直观看到一周内自己把时间都花在了什么地方上。同时也可以很方便的记录时间。 不仅可以管理时间，还可以记录经验，记录灵感，添加倒数日，添加周常事件。
  ☆15Jul 29, 2022Updated 4 years ago
• llmir / The-SUSTech-SYSU-dataset-for-automated-exudate-detection-and-diabetic-retinopathy-grading

  View on GitHub
  Automated detection of exudates from fundus images plays an important role in diabetic retinopathy (DR) screening and evaluation, for whi…
  ☆11Dec 11, 2020Updated 5 years ago
• yangzi0210 / Implementing-precise-image-search-based-on-CLIP-using-text

  View on GitHub
  基于CLIP实现以文精准搜图
  ☆16Sep 20, 2023Updated 3 years ago
• EliasCai / shanghai-citywalk

  View on GitHub
  「城语」APP基于A级景区、历史古迹、文物保护单位等基础数据，利用先进的大模型能力实现智能化的Citywalk 路线规划，包括设计一条路线、生成路线攻略、生成景点的推荐理由等三大核心功能；利用大模型减少了人工编辑和推荐的工作量，并可以根据游客的需求进行个性化定制，提升了游客…
  ☆19Feb 20, 2024Updated 2 years ago
• SongDark / Mecos-tf

  View on GitHub
  Meta-learning-based Cold-Start Sequential Recommendation
  ☆15May 25, 2021Updated 5 years ago