Byte Pair Encoding for Python!
☆232Sep 16, 2022Updated 3 years ago
Alternatives and similar repositories for python-bpe
Users that are interested in python-bpe are comparing it to the libraries listed below
Sorting:
- Calculating Expected Time for training LLM.☆38Apr 17, 2023Updated 2 years ago
- Unsupervised Word Segmentation for Neural Machine Translation and Text Generation☆2,265Aug 7, 2024Updated last year
- KoParadigm: Korean Inflectional Paradigm Generator☆57Nov 23, 2022Updated 3 years ago
- Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE)☆1,221Oct 1, 2024Updated last year
- ☆23Oct 17, 2024Updated last year
- Fast BPE☆679Jun 18, 2024Updated last year
- Memory-efficient Count-Min Sketch Counter (based on Madoka C++ library)☆27Nov 30, 2025Updated 3 months ago
- exBERT on Transformers🤗☆10Jun 14, 2021Updated 4 years ago
- ☆16Sep 14, 2019Updated 6 years ago
- A lightweight but powerful library to build token indices for NLP tasks, compatible with major Deep Learning frameworks like PyTorch and …☆51Dec 6, 2024Updated last year
- A repository comprising of code for generation of noisy speech data from clean data using deep learning methods☆16Jul 12, 2021Updated 4 years ago
- Anh - LAION's multilingual assistant datasets and models☆27Apr 5, 2023Updated 2 years ago
- Jasper 기반 양자화된 모델인 Quartznet 한국어 음성인식☆22Jul 21, 2021Updated 4 years ago
- Getting interpretable dimensions in word embedding spaces.☆15Jul 6, 2023Updated 2 years ago
- Recurrent Neural Network based Hate Speech Language Model for Korean Hate Speech Detection☆23Apr 22, 2020Updated 5 years ago
- Scripts to train a bidirectional LSTM with knowledge distillation from BERT☆159Nov 21, 2019Updated 6 years ago
- Paper: A Simple and Effective Neural Model for Joint Word Segmentation and POS Tagging☆35Mar 7, 2019Updated 7 years ago
- Longformer: The Long-Document Transformer☆2,189Feb 8, 2023Updated 3 years ago
- ALBERT Text Classification Tensorflow, Resume Classification☆15Mar 28, 2020Updated 5 years ago
- Convert Numerical Representations to Korean Pronunciation☆14Apr 20, 2020Updated 5 years ago
- Jejueo Datasets for Machine Translation and Speech Synthesis☆83Feb 19, 2020Updated 6 years ago
- 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.☆81Mar 17, 2022Updated 3 years ago
- Scikit-learn style model finetuning for NLP☆720Oct 21, 2025Updated 4 months ago
- Beyond LM: How can language model go forward in the future?☆15Apr 30, 2023Updated 2 years ago
- annotated-transformer-kr☆15May 16, 2019Updated 6 years ago
- Korean text normalization and language preparation package for LM in Kaldi-based ASR system☆63Apr 23, 2020Updated 5 years ago
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators☆2,370Mar 23, 2024Updated last year
- jiant is an nlp toolkit☆1,674Jul 6, 2023Updated 2 years ago
- Word Piece Model python light version with functions tokenize/save/load☆64Oct 1, 2020Updated 5 years ago
- Dependency Parsing as Sequence Labeling☆27Jul 25, 2024Updated last year
- Code for Context is Everything: Finding Meaning Statistically in Semantic Spaces.☆16Apr 28, 2018Updated 7 years ago
- (re)Implementation of Learning Multi-level Dependencies for Robust Word Recognition☆17Jul 25, 2024Updated last year
- a Pytorch implementation of the Reformer Network (https://openreview.net/pdf?id=rkgNKkHtvB)☆53Nov 22, 2022Updated 3 years ago
- Easy Namuwiki Extractor☆29Nov 29, 2016Updated 9 years ago
- Keras library for building (Universal) Transformers, facilitating BERT and GPT models☆541May 30, 2020Updated 5 years ago
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,677Mar 1, 2026Updated last week
- ☆21May 24, 2023Updated 2 years ago
- Text pattern search using marisa-trie☆18Jan 26, 2025Updated last year
- A toolkit for generating paraphrase vector representations for words in context☆23May 19, 2015Updated 10 years ago