owenliang / bpe-tokenizer
LLM Tokenizer with BPE algorithm
☆31Updated 11 months ago
Alternatives and similar repositories for bpe-tokenizer:
Users that are interested in bpe-tokenizer are comparing it to the libraries listed below
- DeepSpeed Tutorial☆95Updated 8 months ago
- 通义千问的DPO训练☆45Updated 6 months ago
- 一些大语言模型和多模态模型的应用,主要包括Rag，小模型，Agent，跨模态搜索，OCR等等☆161Updated 5 months ago
- ☆64Updated 6 months ago
- ☆107Updated 9 months ago
- 训练一个对中文支持更好的LLaVA模型，并开源训练代码和数据。☆54Updated 7 months ago
- Qwen1.5-SFT(阿里, Ali), Qwen_Qwen1.5-2B-Chat/Qwen_Qwen1.5-7B-Chat微调(transformers)/LORA(peft)/推理☆57Updated 10 months ago
- Build a simple basic multimodal large model from scratch. 从零搭建一个简单的基础多模态大模型🤖☆34Updated 9 months ago
- LLM101n: Let's build a Storyteller 中文版☆130Updated 8 months ago
- Inference code for LLaMA models☆118Updated last year
- 使用单个24G显卡，从0开始训练LLM☆52Updated 5 months ago
- MoE model with onnx runtime☆36Updated 11 months ago
- LLM+RAG for QA☆21Updated last year
- A simple deep learning framework inspired by Dezero and PyTorch☆29Updated 2 months ago
- simple decoder-only GTP model in pytorch☆39Updated 10 months ago
- In this fast-paced world, we all need a little something to spice up life. Whether you need a glass of sweet talk to lift your spirits or…☆52Updated 2 months ago
- This is a detailed code demo on how to conduct Full-Param Supervised Fine-tuning (SFT) and DPO (Direct Preference Optimization)☆12Updated 3 months ago
- ☆29Updated 8 months ago
- 天池算法比赛《BetterMixture - 大模型数据混合挑战赛》的第一名top1解决方案☆28Updated 9 months ago
- 包含程序员面试大厂面试题和面试经验☆124Updated 3 months ago
- 怎么训练一个LLM分词器☆143Updated last year
- an implementation of transformer, bert, gpt, and diffusion models for learning purposes☆152Updated 6 months ago
- ☆77Updated 4 months ago
- ☆60Updated last year
- ☆22Updated last month
- DeepSpeed教程 & 示例注释 & 学习笔记 （大模型高效训练）☆158Updated last year
- 大语言模型应用：RAG、NL2SQL、聊天机器人、预训练、MOE混合专家模型、微调训练、强化学习、天池数据竞赛☆59Updated 2 months ago
- pytorch分布式训练☆65Updated last year
- ☆38Updated last month
- 顾名思义：手搓的RAG☆120Updated last year