bojone / bytepieceLinks
更纯粹、更高压缩率的Tokenizer
☆481Updated 8 months ago
Alternatives and similar repositories for bytepiece
Users that are interested in bytepiece are comparing it to the libraries listed below
Sorting:
- ☆310Updated 2 years ago
- 使用sentencepiece中BPE训练中文词表,并在transformers中进行使用。☆119Updated 2 years ago
- 语言模型中文认知能力分析☆237Updated last year
- 欢迎来到 "LLM-travel" 仓库!探索大语言模型(LLM)的奥秘 🚀。致力于深入理解、探讨以及实现与大模型相关的各种技术、原理和应用。☆335Updated last year
- Collaborative Training of Large Language Models in an Efficient Way☆416Updated 11 months ago
- ☆231Updated last year
- OpenLLMWiki: Docs of OpenLLMAI. Survey, reproduction and domain/task adaptation of open source chatgpt alternatives/implementations. PiXi…☆261Updated 8 months ago
- 中文图书语料MD5链接☆216Updated last year
- BiLLa: A Bilingual LLaMA with Enhanced Reasoning Ability☆418Updated 2 years ago
- 文本去重☆76Updated last year
- Implementation of Chinese ChatGPT☆287Updated last year
- Baichuan2代码的逐行解析版本,适合小白☆214Updated last year
- 中文书籍收录整理, Collection of Chinese Books☆192Updated last year
- ☆175Updated last year
- The Roadmap for LLMs☆86Updated 2 years ago
- alpaca中文指令微调数据集☆394Updated 2 years ago
- 中文 Instruction tuning datasets☆134Updated last year
- Efficient, Low-Resource, Distributed transformer implementation based on BMTrain☆261Updated last year
- 开源SFT数据集整理,随时补充☆535Updated 2 years ago
- 大模型多维度中文对齐评测基准 (ACL 2024)☆407Updated last year
- Naive Bayes-based Context Extension☆326Updated 8 months ago
- 大语言模型指令调优工具( 支持 FlashAttention)☆177Updated last year
- 本项目旨在对大量文本文件进行快速编码检测和转换以辅助mnbvc语料集项目的数据清洗工作☆62Updated 10 months ago
- text embedding☆147Updated last year
- pCLUE: 1000000+多任务提示学习数据集☆500Updated 2 years ago
- The official codes for "Aurora: Activating chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning"☆264Updated last year
- 专注于中文领域大语言模型,落地到某个行业某个领域,成为一个行业 大模型、公司级别或行业级别领域大模型。☆120Updated 5 months ago
- ☆460Updated last year
- Firefly中文LLaMA-2大模型,支持增量预训练Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom等大模型☆413Updated last year
- ☆281Updated last year