owenliang / bpe-tokenizerLinks
LLM Tokenizer with BPE algorithm
☆32Updated last year
Alternatives and similar repositories for bpe-tokenizer
Users that are interested in bpe-tokenizer are comparing it to the libraries listed below
Sorting:
- 通义千问的DPO训练☆48Updated 9 months ago
- simple decoder-only GTP model in pytorch☆41Updated last year
- WWW2025 Multimodal Intent Recognition for Dialogue Systems Challenge☆120Updated 7 months ago
- ☆82Updated 8 months ago
- 训练一个对中文支持更好的LLaVA模型,并开源训练代码和数据。☆61Updated 9 months ago
- Inference code for LLaMA models☆121Updated last year
- ☆111Updated 11 months ago
- 一些大语言模型和多模态模型的生态,主要包括跨模态搜索、投机解码、QAT量化、多模态量化、ChatBot、OCR☆182Updated this week
- Pytorch DDP Traning Demo☆24Updated 8 months ago
- Build a simple basic multimodal large model from scratch. 从零搭建一个简单的基础多模态大模型🤖☆41Updated last year
- LLM101n: Let's build a Storyteller 中文版☆131Updated 10 months ago
- ThinkLLM:🚀 轻量、高效的大语言模型算法实现☆73Updated last month
- LLM+RAG for QA☆22Updated last year
- DeepSpeed教程 & 示例注释 & 学习笔记 (大模型高效训练)☆169Updated last year
- Qwen1.5-SFT(阿里, Ali), Qwen_Qwen1.5-2B-Chat/Qwen_Qwen1.5-7B-Chat微调(transformers)/LORA(peft)/推理☆63Updated last year
- DeepSpeed Tutorial☆97Updated 10 months ago
- ☆41Updated 3 months ago
- A simple deep learning framework inspired by Dezero and PyTorch☆29Updated 4 months ago
- This is a detailed code demo on how to conduct Full-Param Supervised Fine-tuning (SFT) and DPO (Direct Preference Optimization)☆14Updated 5 months ago
- 天池算法比赛《BetterMixture - 大模型数据混合挑战赛》的第一名top1解决方案☆31Updated 11 months ago
- ☆43Updated 10 months ago
- 本项目用于Embedding模型的相关实验,包括Embedding模型评估、Embedding模型微调、Embedding模型量化等。☆55Updated 11 months ago
- ☆72Updated last month
- ☆109Updated 7 months ago
- ☆63Updated last year
- ☆31Updated 10 months ago
- 使用单个24G显卡,从0开始训练LLM☆55Updated last month
- ☆85Updated last week
- 快速入门RAG与私有化部署☆191Updated last year
- 基于大语言模型的检索增强生成RAG示例☆151Updated last month