怎么训练一个LLM分词器
☆152Jul 13, 2023Updated 3 years ago
Alternatives and similar repositories for how-to-train-tokenizer
Users that are interested in how-to-train-tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- GoGPT:基于Llama/Llama 2训练的中英文增强大模型|Chinese-Llama2☆78Oct 7, 2023Updated 3 years ago
- 使用单个24G显卡，从0开始训练LLM☆54Jul 9, 2025Updated last year
- 欢迎来到 "LLM-travel" 仓库！探索大语言模型（LLM）的奥秘 🚀。致力于深入理解、探讨以及实现与大模型相关的各种技术、原理和应用。☆385Jul 21, 2024Updated 2 years ago
- ☆307Apr 6, 2023Updated 3 years ago
- BLOOM 模型的指令微调☆24Jun 15, 2023Updated 3 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆118Jun 24, 2023Updated 3 years ago
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆19Jul 20, 2023Updated 3 years ago
- 语言模型中文认知能力分析☆235Sep 9, 2023Updated 3 years ago
- Firefly中文LLaMA-2大模型，支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型☆415Oct 21, 2023Updated 2 years ago
- Chinese-LLaMA 1&2、Chinese-Falcon 基础模型；ChatFlow中文对话模型；中文OpenLLaMA模型；NLP预训练/指令微调数据集☆3,036Apr 14, 2024Updated 2 years ago
- 中文nlp解决方案(大模型、数据、模型、训练、推理)☆3,842Aug 5, 2025Updated last year
- ☆26Dec 2, 2022Updated 3 years ago
- 中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。☆1,740Apr 20, 2024Updated 2 years ago
- 基于 onnxruntime 推理引擎的中文 ltp 词法分析☆14Oct 4, 2022Updated 4 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- 用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.☆2,922Aug 13, 2026Updated 2 months ago
- Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo☆1,094Aug 4, 2024Updated 2 years ago
- Debug DeepSpeed-Chat step by step in IDE (在IDE里一步一步调试DeepSpeed-Chat)☆10Apr 17, 2023Updated 3 years ago
- Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、…☆6,652Oct 24, 2024Updated last year
- [COLING 2022] CSL: A Large-scale Chinese Scientific Literature Dataset 中文科学文献数据集☆674Jun 19, 2023Updated 3 years ago
- Train a 1B LLM with 1T tokens from scratch by personal☆818Apr 27, 2025Updated last year
- Chinese large language model☆123Jun 29, 2023Updated 3 years ago
- Deepspeed、LLM、Medical_Dialogue、医疗大模型、预训练、微调☆317May 3, 2026Updated 5 months ago
- Large-scale exact string matching tool☆17Mar 7, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆164Apr 17, 2023Updated 3 years ago
- EMNLP 2021: A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding☆10Apr 8, 2022Updated 4 years ago
- ChatGLM2-6B-Explained☆36Jul 28, 2023Updated 3 years ago
- ☆367Jun 13, 2024Updated 2 years ago
- Easy and Efficient Finetuning LLMs. (Supported LLama, LLama2, LLama3, Qwen, Baichuan, GLM , Falcon) 大模型高效量化训练+部署.☆623Jan 24, 2025Updated last year
- Baichuan2代码的逐行解析版本，适合小白☆210Sep 20, 2023Updated 3 years ago
- Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。☆1,227Feb 27, 2024Updated 2 years ago
- pCLUE: 1000000+多任务提示学习数据集☆508Oct 4, 2022Updated 4 years ago
- TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLO…☆981Sep 14, 2024Updated 2 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- 一个强调工程化、可观测、可测试、可扩展的 RAG 项目。TraceRAG 的目标不是只把答案“生成出来”，而是把文 档导入、切块、向量化、检索、带来源回答、评估与后续 tracing 拆成可独立验证的阶段，逐步演进成一个可维护、可解释、可复盘的生产级 RAG。☆15Apr 2, 2026Updated 6 months ago
- Manages vllm-nccl dependency☆19Jun 3, 2024Updated 2 years ago
- An open-source conversational language model developed by the Knowledge Works Research Laboratory at Fudan University.☆64Oct 12, 2023Updated 3 years ago
- 基于ChatGLM-6B + LoRA的Fintune方案☆3,738Nov 25, 2023Updated 2 years ago
- 基于ChatGLM-6B、ChatGLM2-6B、ChatGLM3-6B模型，进行下游具体任务微调，涉及Freeze、Lora、P-tuning、全参微调等☆2,773Dec 12, 2023Updated 2 years ago
- ☆15Aug 21, 2023Updated 3 years ago
- A large-scale 7B pretraining language model developed by BaiChuan-Inc.☆5,649Jul 18, 2024Updated 2 years ago