zejunwang1 / easytokenizer
高性能文本 Tokenizer 库
☆28Updated last year
Alternatives and similar repositories for easytokenizer:
Users that are interested in easytokenizer are comparing it to the libraries listed below
- BERT Tokenizer in C++☆75Updated 4 years ago
- 大规模中文语料☆40Updated 5 years ago
- 一个非常高效的字符串匹配工具，支持正向/反向最大匹配分词和多模式字符串精确匹配☆17Updated last year
- Chinese MobileBERT（中文MobileBERT模型）☆89Updated 3 years ago
- 百川Dynamic NTK-ALiBi的代码实现：无需微调即可推理更长文本☆47Updated last year
- A more efficient GLM implementation!☆55Updated 2 years ago
- 纯c++的全平台llm加速库，支持python调用，支持baichuan, glm, llama, moss基座，手机端流畅运行chatglm-6B级模型单卡可达10000+token / s，☆45Updated last year
- 一个基于预训练的句向量生成工具☆136Updated last year
- ☆90Updated last year
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆116Updated last year
- llama inference for tencentpretrain☆98Updated last year
- 介绍docker、docker compose的使用。☆20Updated 6 months ago
- CTC2021-中文文本纠错大赛的SOTA方案及在线演示☆72Updated last year
- text embedding☆144Updated last year
- LORA微调BLOOMZ，参考BELLE☆25Updated 2 years ago
- 文本去重☆69Updated 10 months ago
- baichuan LLM surpervised finetune by lora☆62Updated last year
- sentence-transformers to onnx 让sbert模型推理效率更快☆163Updated 3 years ago
- ChatGLM2-6B微调, SFT/LoRA, instruction finetune☆106Updated last year
- 基于模板的文本纠错；Automatically Mining Error Templates for Grammatical Error Correction☆39Updated 2 years ago
- CLUEWSC2020: WSC Winograd模式挑战中文版，中文指代消解任务☆74Updated 4 years ago
- Correcting Chinese Spelling Errors with Phonetic Pre-training 非官方实现☆40Updated 3 years ago
- 中文 Instruction tuning datasets☆129Updated 11 months ago
- BLOOM 模型的指令微调☆24Updated last year
- 本项目旨在对大量文本文件进行快速编码检测和转换以辅助mnbvc语料集项目的数据清洗工作☆60Updated 5 months ago
- ☆84Updated last year
- 中文版unilm预训练模型☆83Updated 4 years ago
- 通用版面分析 | 中文文档解析 |Document Layout Analysis | layout paser☆46Updated 9 months ago
- A repo for update and debug Mixtral-7x8B、MOE、ChatGLM3、LLaMa2、 BaChuan、Qwen an other LLM models include new models mixtral, mixtral 8x7b, …☆43Updated last week
- moss chat finetuning☆50Updated 11 months ago