怎么训练一个LLM分词器
☆152Jul 13, 2023Updated 3 years ago
Alternatives and similar repositories for how-to-train-tokenizer
Users that are interested in how-to-train-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- GoGPT:基于Llama/Llama 2训练的中英文增强大模型|Chinese-Llama2☆78Oct 7, 2023Updated 3 years ago
- 使用单个24G显卡，从0开始训练LLM☆54Jul 9, 2025Updated last year
- 欢迎来到 "LLM-travel" 仓库！探索大语言模型（LLM）的奥秘 🚀。致力于深入理解、探讨以及实现与大模型相关的各种技术、原理和应用。☆385Jul 21, 2024Updated 2 years ago
- ☆307Apr 6, 2023Updated 3 years ago
- BLOOM 模型的指令微调☆24Jun 15, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆118Jun 24, 2023Updated 3 years ago
- Chinese large language model☆123Jun 29, 2023Updated 3 years ago
- Firefly中文LLaMA-2大模型，支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型☆415Oct 21, 2023Updated 3 years ago
- 语言模型中文认知能力分析☆235Sep 9, 2023Updated 3 years ago
- 中文nlp解决方案(大模型、数据、模型、训练、推理)☆3,843Aug 5, 2025Updated last year
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆19Jul 20, 2023Updated 3 years ago
- Chinese-LLaMA 1&2、Chinese-Falcon 基础模型；ChatFlow中文对话模型；中文OpenLLaMA模型；NLP预训练/指令微调数据集☆3,037Apr 14, 2024Updated 2 years ago
- 中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来 源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。☆1,737Apr 20, 2024Updated 2 years ago
- 用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.☆2,926Aug 13, 2026Updated 2 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- ☆26Dec 2, 2022Updated 3 years ago
- 基于 onnxruntime 推理引擎的中文 ltp 词法分析☆14Oct 4, 2022Updated 4 years ago
- Train a 1B LLM with 1T tokens from scratch by personal☆819Apr 27, 2025Updated last year
- Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、…☆6,642Oct 24, 2024Updated 2 years ago
- Debug DeepSpeed-Chat step by step in IDE (在IDE里一步一步调试DeepSpeed-Chat)☆10Apr 17, 2023Updated 3 years ago
- Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo☆1,096Aug 4, 2024Updated 2 years ago
- 2023全球智能汽车AI挑战赛——赛道一：AI大模型检索问答， 75+ baseline☆60Dec 7, 2023Updated 2 years ago
- Easy and Efficient Finetuning LLMs. (Supported LLama, LLama2, LLama3, Qwen, Baichuan, GLM , Falcon) 大模型高效量化训练+部署.☆624Jan 24, 2025Updated last year
- “阿里灵杰”问天引擎电商搜索算法赛 第二名。电商领域两阶段文本匹配算法。☆56Jul 28, 2022Updated 4 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLO…☆978Sep 14, 2024Updated 2 years ago
- [COLING 2022] CSL: A Large-scale Chinese Scientific Literature Dataset 中文科学文献数据集☆671Jun 19, 2023Updated 3 years ago
- [NeurIPS 2024 Main Track] Code for the paper titled "Instruction Tuning With Loss Over Instructions"☆38May 24, 2024Updated 2 years ago
- Large-scale exact string matching tool☆17Mar 7, 2025Updated last year
- pCLUE: 1000000+多任务提示学习数据集☆508Oct 4, 2022Updated 4 years ago
- EMNLP 2021: A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding☆10Apr 8, 2022Updated 4 years ago
- Tuning LLMs with no tears💦; Sample Design Engineering (SDE) for more efficient downstream-tuning.☆1,019Apr 27, 2024Updated 2 years ago
- BELLE: Be Everyone's Large Language model Engine（开源中文对话大模型）☆8,261Oct 16, 2024Updated 2 years ago
- ⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification, Text-Generation, Information-Extraction, Text-Matching, RLHF, SF…☆2,427Sep 29, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志…☆4,298Oct 25, 2026Updated 2 weeks ago
- ChatGLM2-6B-Explained☆36Jul 28, 2023Updated 3 years ago
- 基于ChatGLM-6B、ChatGLM2-6B、ChatGLM3-6B模型，进行下游具体任务微调，涉及Freeze、Lora、P-tuning、全参微调等☆2,773Dec 12, 2023Updated 2 years ago
- ☆163Apr 17, 2023Updated 3 years ago
- 基于ChatGLM-6B + LoRA的Fintune方案☆3,736Nov 25, 2023Updated 2 years ago
- Baichuan2代码的逐行解析版本，适合小白☆210Sep 20, 2023Updated 3 years ago
- A large-scale 7B pretraining language model developed by BaiChuan-Inc.☆5,645Jul 18, 2024Updated 2 years ago