yanqiangmiffy / how-to-train-tokenizerView external linksLinks
怎么训练一个LLM分词器
☆153Jul 13, 2023Updated 2 years ago
Alternatives and similar repositories for how-to-train-tokenizer
Users that are interested in how-to-train-tokenizer are comparing it to the libraries listed below
Sorting:
- GoGPT:基于Llama/Llama 2训 练的中英文增强大模型|Chinese-Llama2☆78Oct 7, 2023Updated 2 years ago
- ☆313Apr 6, 2023Updated 2 years ago
- BLOOM 模型的指令微调☆24Jun 15, 2023Updated 2 years ago
- 欢迎来到 "LLM-travel" 仓库！探索大语言模型（LLM）的奥秘 🚀。致力于深入理解、探讨以及实现与大模型相关的各种技术、原理和应用。☆372Jul 21, 2024Updated last year
- 使用单个24G显卡，从0开始训练LLM☆56Jul 9, 2025Updated 7 months ago
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆120Jun 24, 2023Updated 2 years ago
- ☆26Dec 2, 2022Updated 3 years ago
- 语言模型中文认知能力分析☆235Sep 9, 2023Updated 2 years ago
- Chinese-LLaMA 1&2、Chinese-Falcon 基础模型；ChatFlow中文对话模型；中文OpenLLaMA模型；NLP预训练/指令微调数据集☆3,055Apr 14, 2024Updated last year
- Firefly中文LLaMA-2大模型，支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型☆416Oct 21, 2023Updated 2 years ago
- 中文nlp解决方案(大模型、数据、模型、训练、推理)☆3,777Aug 5, 2025Updated 6 months ago
- Deepspeed、LLM、Medical_Dialogue、医疗大模型、预训练、微调☆289Jun 7, 2024Updated last year
- 中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。☆1,672Apr 20, 2024Updated last year
- [NeurIPS 2024 Main Track] Code for the paper titled "Instruction Tuning With Loss Over Instructions"☆38May 24, 2024Updated last year
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆19Jul 20, 2023Updated 2 years ago
- Train a 1B LLM with 1T tokens from scratch by personal☆788Apr 27, 2025Updated 9 months ago
- [COLING 2022] CSL: A Large-scale Chinese Scientific Literature Dataset 中文科学文献数据集☆660Jun 19, 2023Updated 2 years ago
- Baichuan2代码的逐行解析版本，适合小白☆212Sep 20, 2023Updated 2 years ago
- Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、…☆6,635Oct 24, 2024Updated last year
- 🛰️ 基于真实医疗对话数据在ChatGLM上进行LoRA、P-Tuning V2、Freeze、RLHF等微调，我们的眼光不止于医疗问答☆337Sep 2, 2023Updated 2 years ago
- Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo☆1,089Aug 4, 2024Updated last year
- The official github repo for the open online courses: "Dive into LLMs".☆10Mar 15, 2024Updated last year
- Debug DeepSpeed-Chat step by step in IDE (在IDE里一步一步调试DeepSpeed-Chat)☆10Apr 17, 2023Updated 2 years ago
- Easy and Efficient Finetuning LLMs. (Supported LLama, LLama2, LLama3, Qwen, Baichuan, GLM , Falcon) 大模型高效量化训练+部署.☆619Jan 24, 2025Updated last year
- Implementation of Chinese ChatGPT☆288Nov 20, 2023Updated 2 years ago
- 用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.☆2,889May 21, 2024Updated last year
- 2023全球智能汽车AI挑战赛——赛道一：AI大模型检索问答， 75+ baseline☆60Dec 7, 2023Updated 2 years ago
- ☆164Apr 17, 2023Updated 2 years ago
- An open-source conversational language model developed by the Knowledge Works Research Laboratory at Fudan University.☆64Oct 12, 2023Updated 2 years ago
- Long Context Research☆26Jan 26, 2026Updated 3 weeks ago
- 刹那是永恒☆13Feb 26, 2020Updated 5 years ago
- ☆28Jan 5, 2026Updated last month
- TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLO…☆981Sep 14, 2024Updated last year
- Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。☆1,127Feb 27, 2024Updated last year
- NTK scaled version of ALiBi position encoding in Transformer.☆69Aug 16, 2023Updated 2 years ago
- 基于ChatGLM-6B、ChatGLM2-6B、ChatGLM3-6B模型，进行下游具体任务微调，涉及Freeze、Lora、P-tuning、全参微调等☆2,776Dec 12, 2023Updated 2 years ago
- ⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification, Text-Generation, Information-Extraction, Text-Matching, RLHF, SF…☆2,409Sep 29, 2023Updated 2 years ago
- ☆363Jun 13, 2024Updated last year
- open-deepsearch☆11Mar 3, 2025Updated 11 months ago