howl-anderson / MicroTokenizer
一个轻量且功能全面的中文分词器,帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..
☆150Updated 5 months ago
Alternatives and similar repositories for MicroTokenizer:
Users that are interested in MicroTokenizer are comparing it to the libraries listed below
- Self complemented Pinyin2Chinese demo use algorithms including Trie and HMM model , 基于隐马尔科夫模型与Trie树的拼音切分与拼音转中文的简单demo实现。☆86Updated 6 years ago
- THU Chinese Keyphrase Extraction Toolkit☆125Updated 6 years ago
- Train Wikidata with word2vec for word embedding tasks☆122Updated 6 years ago
- 基于最小熵原理的NLP工具包☆138Updated 3 years ago
- ChineseTextualInference project including chinese corpus build and inferecence model, 中文文本推断项目,包括88万文本蕴含中文文本蕴含数据集的翻译与构建,基于深度学习的文本蕴含判定模型构建…☆172Updated 6 years ago
- 中文相关词典和语料库。☆173Updated 10 years ago
- This is a corpus of Chinese abbreviation, including negative full forms.☆194Updated 3 years ago
- 简易的中文纠错和消歧☆289Updated 9 years ago
- NLU is hard!!!☆273Updated 6 years ago
- 中文生成式预训练模型☆98Updated 4 years ago
- 各大中文分词性能评测☆157Updated 6 years ago
- 基于哈工大同义词词林扩展版的单词相似度计算方法☆364Updated last year
- 中文文本自动纠错☆84Updated 6 years ago
- 一个中文的已标注词性的语料库☆201Updated 10 years ago
- 依存关系分析,NLP,自然语言处理☆85Updated 3 years ago
- 汉字字符特征提取器 (featurizer),提取汉字的特征(发音特征、字形特征)用做深度学习的特征 | A Chinese character feature extractor, which extracts the features of Chinese charac…☆293Updated 4 years ago
- 中文分词工具评估☆61Updated 2 years ago
- SemEval-2016 Task 9: Chinese Semantic Dependency Parsing☆135Updated 6 years ago
- A Chinese sentiment dataset may be useful for sentiment analysis.☆231Updated 8 years ago
- Simple Solution for Multi-Criteria Chinese Word Segmentation☆301Updated 4 years ago
- Code for chinese error detection module, using n-gram and bi-lstm☆135Updated 5 years ago
- 李傲龍的博客☆81Updated 8 months ago
- An collection of Chinese nlp corpus including basic Chinese syntatic wordset, semantic wordset, historic corpus and evaluate corpus. 中文自然…☆443Updated 6 years ago
- NLP NER datasets video/music/book bio☆88Updated 4 years ago
- Macadam是一个以Tensorflow(Keras)和bert4keras为基础,专注于文本分类、序列标注和关系抽取的自然语言处理工具包。支持RANDOM、WORD2VEC、FASTTEXT、BERT、ALBERT、ROBERTA、NEZHA、XLNET、ELECTRA…☆324Updated 2 years ago
- self complemented SpellCorrection based pinyin similairity, edit distance ,基于拼音相似度与编辑距离的查询纠错。☆82Updated 2 years ago
- Some useful Chinese corpus datasets 中文语料小数据☆533Updated 5 years ago
- HyponymyExtraction and Graph based on KB Schema, Baike-kb and online text extract, 基于知识概念体系,百科知识库,以及在线搜索结构化方式的词语上下位抽取与可视化展示☆170Updated 6 years ago
- 人民日报语料处理工具集 | Tools for Corpus of People's Daily☆278Updated last year
- Chinese Language Generation Evaluation 中文生成任务基准测评☆245Updated 4 years ago