zejunwang1 / easytokenizerLinks
高性能文本 Tokenizer 库
☆29Updated last year
Alternatives and similar repositories for easytokenizer
Users that are interested in easytokenizer are comparing it to the libraries listed below
Sorting:
- 一个非常高效的字符串匹配工具，支持正向/反向最大匹配分词和多模式字符串精确匹配☆17Updated last year
- A more efficient GLM implementation!☆55Updated 2 years ago
- ☆90Updated last year
- 纯c++的全平台llm加速库，支持python调用，支持baichuan, glm, llama, moss基座，手机端流畅运行chatglm-6B级模型单卡可达10000+token / s，☆45Updated last year
- BERT Tokenizer in C++☆76Updated 4 years ago
- 一个基于预训练的句向量生成工具☆137Updated 2 years ago
- 大规模中文语料☆42Updated 5 years ago
- 百川Dynamic NTK-ALiBi的代码实现：无需微调即可推理更长文本☆47Updated last year
- baichuan LLM surpervised finetune by lora☆63Updated last year
- sentence-transformers to onnx 让sbert模型推理效率更快☆164Updated 3 years ago
- Another ChatGLM2 implementation for GPTQ quantization☆54Updated last year
- GoGPT:基于Llama/Llama 2训练的中英文增强大模型|Chinese-Llama2☆78Updated last year
- 通用版面分析 | 中文文档解析 |Document Layout Analysis | layout paser☆46Updated last year
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆69Updated last year
- CLUEWSC2020: WSC Winograd模式挑战中文版，中文指代消解任务☆75Updated 5 years ago
- 中文 Instruction tuning datasets☆132Updated last year
- llama inference for tencentpretrain☆98Updated 2 years ago
- text embedding☆146Updated last year
- code and data for "CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers"☆69Updated 10 months ago
- lightweighted deep learning inference service framework☆39Updated 4 years ago
- Correcting Chinese Spelling Errors with Phonetic Pre-training 非官方实现☆40Updated 3 years ago
- 文本去重☆72Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMs☆135Updated 6 months ago
- A repo for update and debug Mixtral-7x8B、MOE、ChatGLM3、LLaMa2、 BaChuan、Qwen an other LLM models include new models mixtral, mixtral 8x7b, …☆46Updated last week
- Code & Data for our Paper "NaSGEC: Multi-Domain Chinese Grammatical Error Correction for Native Speaker Texts" (ACL 2023 Findings)☆89Updated 4 months ago
- CTC2021-中文文本纠错大赛的SOTA方案及在线演示☆72Updated last year
- A Multi-modal Model Chinese Spell Checker Released on ACL2021.☆159Updated last year
- 使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。☆118Updated 2 years ago
- Minimal keyword extraction with BERT☆84Updated 3 years ago
- Source code for the paper "PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction" in ACL2021☆236Updated 2 years ago