uhermjakob / utokenLinks
universal tokenizer
☆17Updated 3 years ago
Alternatives and similar repositories for utoken
Users that are interested in utoken are comparing it to the libraries listed below
Sorting:
- A python module for word inflections designed for use with spaCy.☆93Updated 5 years ago
- Bilingual sentence similarity classifier using Tensorflow☆23Updated 5 years ago
- This packages up data for the Open Multilingual Wordnet☆52Updated 3 months ago
- A set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor/minority languages.☆36Updated this week
- Scripts investigate, repair and normalize a wide range of text file problems at the character level.☆19Updated 2 years ago
- Wiktra - Python tool of Wiktionary Transliteration modules for 514 languages and its 102 different scripts (orthographies)☆30Updated 2 months ago
- Efficient Low-Memory Aligner☆146Updated 7 months ago
- Curated corpus of parallel data derived from versions of the Bible provided by eBible.org.☆70Updated 3 months ago
- Character-level conversion between Hebrew text and Latin transliteration using deep learning - a demonstration of seq2seq training.☆14Updated 2 years ago
- Python framework for processing Universal Dependencies data☆57Updated 2 weeks ago
- Text tokenization and sentence segmentation (segtok v2)☆205Updated 3 years ago
- Master repo for the UniMorph project, includes the UniMorph schema and annotated data files☆32Updated 5 years ago
- Python package for WikiMedia dump processing (Wiktionary, Wikipedia etc). Wikitext parsing, template expansion, Lua module execution. Fo…☆106Updated this week
- Transform TMX to text☆27Updated 2 years ago
- OpusFilter - Parallel corpus processing toolkit☆109Updated 3 weeks ago
- Sentence aligner☆116Updated 4 years ago
- A python module for English lemmatization and inflection.☆270Updated last year
- Efficient teacher-student models and scripts to make them☆51Updated last year
- A python true casing utility that restores case information for texts☆89Updated 2 years ago
- MorphyNet: a Large Multilingual Database of Derivational and Inflectional Morphology (+morpheme segmentation)☆47Updated 2 years ago
- Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.☆158Updated last year
- Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.☆252Updated 2 years ago
- A single model that parses Universal Dependencies across 75 languages. Given a sentence, jointly predicts part-of-speech tags, morphology…☆223Updated 2 years ago
- An easy-to-use library to linguistically compare one sentence and its words to another, in the same language or a different one. For inst…☆24Updated 3 years ago
- A sentence segmentation library with wide language support optimized for speed and utility.☆66Updated 2 months ago
- A list of vocabulary lists☆22Updated 5 years ago
- Simple multilingual lemmatizer for Python, especially useful for speed and efficiency☆171Updated 2 months ago
- Bitextor generates translation memories from multilingual websites☆295Updated 9 months ago
- Python Finite-State Toolkit☆58Updated this week
- Various utilities for processing the data.☆211Updated last week