zejunwang1 / easytokenizerLinks
高性能文本 Tokenizer 库
☆29Updated last year
Alternatives and similar repositories for easytokenizer
Users that are interested in easytokenizer are comparing it to the libraries listed below
Sorting:
- 一个非常高效的字符串匹配工具，支持正向/反向最大匹配分词和多模式字符串精确匹配☆17Updated last year
- 纯c++的全平台llm加速库，支持python调用，支持baichuan, glm, llama, moss基座，手机端流畅运行chatglm-6B级模型单卡可达10000+token / s，☆45Updated last year
- 百川Dynamic NTK-ALiBi的代码实现：无需微调即可推理更长文本☆47Updated last year
- A more efficient GLM implementation!☆55Updated 2 years ago
- 一个基于预训练的句向量生成工具☆136Updated 2 years ago
- 大规模中文语料☆42Updated 5 years ago
- ☆90Updated last year
- BERT Tokenizer in C++☆76Updated 4 years ago
- CLUEWSC2020: WSC Winograd模式挑战中文版，中文指代消解任务☆75Updated 5 years ago
- sentence-transformers to onnx 让sbert模型推理效率更快☆164Updated 3 years ago
- 文本去重☆72Updated last year
- 本项目旨在对大量文本文件进行快速编码检测和转换以辅助mnbvc语料集项目的数据清洗工作☆61Updated 7 months ago
- GoGPT:基于Llama/Llama 2训练的中英文增强大模型|Chinese-Llama2☆78Updated last year
- llama inference for tencentpretrain☆98Updated last year
- CTC2021-中文文本纠错大赛的SOTA方案及在线演示☆72Updated last year
- baichuan LLM surpervised finetune by lora☆63Updated last year
- 基于模板的文本纠错；Automatically Mining Error Templates for Grammatical Error Correction☆40Updated 2 years ago
- Chinese MobileBERT（中文MobileBERT模型）☆92Updated 3 years ago
- A Multi-modal Model Chinese Spell Checker Released on ACL2021.☆159Updated last year
- Another ChatGLM2 implementation for GPTQ quantization☆53Updated last year
- text embedding☆146Updated last year
- moss chat finetuning☆50Updated last year
- large language model training-3-stages+deployment☆48Updated last year
- ☆84Updated last year
- LORA微调BLOOMZ，参考BELLE☆25Updated 2 years ago
- ChatGLM2-6B微调, SFT/LoRA, instruction finetune☆108Updated last year
- TensorRT☆11Updated 4 years ago
- 介绍docker、docker compose的使用。☆20Updated 9 months ago
- code for piccolo embedding model from SenseTime☆126Updated last year
- Correcting Chinese Spelling Errors with Phonetic Pre-training 非官方实现☆40Updated 3 years ago