怎么训练一个LLM分词器
☆152Jul 13, 2023Updated 2 years ago
Alternatives and similar repositories for how-to-train-tokenizer
Users that are interested in how-to-train-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- GoGPT:基于Llama/Llama 2训练的中英文增强大模型|Chinese-Llama2☆78Oct 7, 2023Updated 2 years ago
- 欢迎来到 "LLM-travel" 仓库！探索大语言模型（LLM）的奥秘 🚀。致力于深入理解、探讨以及实现与大模型相关的各种技术、原理和应用。☆377Jul 21, 2024Updated last year
- 使用单个24G显卡，从0开始训练LLM☆55Jul 9, 2025Updated 8 months ago
- ☆313Apr 6, 2023Updated 2 years ago
- BLOOM 模型的指令微调☆24Jun 15, 2023Updated 2 years ago
- DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆120Jun 24, 2023Updated 2 years ago
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆19Jul 20, 2023Updated 2 years ago
- 语言模型中文认知能力分析☆236Sep 9, 2023Updated 2 years ago
- Chinese-LLaMA 1&2、Chinese-Falcon 基础模型；ChatFlow中文对话模型；中文OpenLLaMA模型；NLP预训练/指令微调数据集☆3,054Apr 14, 2024Updated last year
- ☆26Dec 2, 2022Updated 3 years ago
- Firefly中文LLaMA-2大模型，支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型☆416Oct 21, 2023Updated 2 years ago
- 中文nlp解决方案(大模型、数据、模型、训练、推理)☆3,789Aug 5, 2025Updated 7 months ago
- 中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支 持下游任务sft微调，给出三元组信息抽取微调示例。☆1,687Apr 20, 2024Updated last year
- 基于 onnxruntime 推理引擎的中文 ltp 词法分析☆14Oct 4, 2022Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting with the flexibility to host WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Cloudways by DigitalOcean.
- Debug DeepSpeed-Chat step by step in IDE (在IDE里一步一步调试DeepSpeed-Chat)☆10Apr 17, 2023Updated 2 years ago
- 用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.☆2,903May 21, 2024Updated last year
- Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、…☆6,652Oct 24, 2024Updated last year
- [NeurIPS 2024 Main Track] Code for the paper titled "Instruction Tuning With Loss Over Instructions"☆38May 24, 2024Updated last year
- Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo☆1,089Aug 4, 2024Updated last year
- The official github repo for the open online courses: "Dive into LLMs".☆10Mar 15, 2024Updated 2 years ago
- Deepspeed、LLM、Medical_Dialogue、医疗大模型、预训练、微调☆296Jun 7, 2024Updated last year
- Train a 1B LLM with 1T tokens from scratch by personal☆792Apr 27, 2025Updated 11 months ago
- Large-scale exact string matching tool☆17Mar 7, 2025Updated last year
- NordVPN Special Discount Offer • AdSave on top-rated NordVPN 1 or 2-year plans with secure browsing, privacy protection, and support for for all major platforms.
- [COLING 2022] CSL: A Large-scale Chinese Scientific Literature Dataset 中文科学文献数据集☆663Jun 19, 2023Updated 2 years ago
- ChatGLM2-6B-Explained☆36Jul 28, 2023Updated 2 years ago
- ☆164Apr 17, 2023Updated 2 years ago
- ☆364Jun 13, 2024Updated last year
- EMNLP 2021: A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding☆10Apr 8, 2022Updated 3 years ago
- Easy and Efficient Finetuning LLMs. (Supported LLama, LLama2, LLama3, Qwen, Baichuan, GLM , Falcon) 大模型高效量化训练+部署.☆620Jan 24, 2025Updated last year
- Baichuan2代码的逐行解析版本，适合小白☆212Sep 20, 2023Updated 2 years ago
- Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。☆1,142Feb 27, 2024Updated 2 years ago
- pCLUE: 1000000+多任务提示学习数据集☆508Oct 4, 2022Updated 3 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLO…☆980Sep 14, 2024Updated last year
- ☆22Jul 1, 2024Updated last year
- Manages vllm-nccl dependency☆17Jun 3, 2024Updated last year
- 使用Sentencepiece对中文语料进行分词☆13Nov 30, 2023Updated 2 years ago
- 基于ChatGLM-6B、ChatGLM2-6B、ChatGLM3-6B模型，进行下游具体任务微调，涉及Freeze、Lora、P-tuning、全参微调等☆2,782Dec 12, 2023Updated 2 years ago
- An open-source conversational language model developed by the Knowledge Works Research Laboratory at Fudan University.☆64Oct 12, 2023Updated 2 years ago
- ☆15Aug 21, 2023Updated 2 years ago