uhermjakob / utoken
universal tokenizer
☆15Updated 3 years ago
Alternatives and similar repositories for utoken:
Users that are interested in utoken are comparing it to the libraries listed below
- English Resource Grammar☆20Updated 6 months ago
- bilingual dictionary extractor from parallel corpora☆22Updated 10 years ago
- A python module for word inflections designed for use with spaCy.☆92Updated 5 years ago
- Efficient teacher-student models and scripts to make them☆49Updated last year
- Python framework for processing Universal Dependencies data☆55Updated 2 weeks ago
- An easy-to-use library to linguistically compare one sentence and its words to another, in the same language or a different one. For inst…☆22Updated 3 years ago
- Code for SaGe subword tokenizer (EACL 2023)☆22Updated 2 months ago
- Efficient Low-Memory Aligner☆141Updated last month
- Python Finite-State Toolkit☆50Updated last month
- Bilingual sentence similarity classifier using Tensorflow☆20Updated 5 years ago
- Extracts plain text, language identification and more metadata from WARC records☆21Updated 2 weeks ago
- A python library / model for creating co-references between AMR graph nodes.☆9Updated 2 years ago
- Transform TMX to text☆28Updated 2 years ago
- A simple configurable tool for manipulating dependency trees.☆13Updated last month
- Scripts investigate, repair and normalize a wide range of text file problems at the character level.☆18Updated 2 years ago
- Framework for creating and accessing UBY resources – sense-linked lexical resources in standard UBY-LMF format☆22Updated 6 years ago
- UFSAC is a resource containing all WordNet Sense Annotated Corpora, and a Java library for manipulating them☆37Updated 2 years ago
- Finds linguistic patterns effortlessly☆35Updated last year
- OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.☆49Updated last month
- Master repo for the UniMorph project, includes the UniMorph schema and annotated data files☆26Updated 5 years ago
- Python libraries for DELPH-IN☆82Updated 5 months ago
- WordNet-LMF formats☆21Updated last week
- Automatic extraction of edited sentences from text edition histories.☆82Updated 3 years ago
- Measure the similarity of text corpora for 74 languages☆13Updated last year
- FoLiA Linguistic Annotation Tool -- Flat is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.g…☆111Updated 3 weeks ago
- ParaNames: A multilingual resource for parallel names☆30Updated 9 months ago
- FoLiA: Format for Linguistic Annotation - FoLiA is a rich XML-based annotation format for the representation of language resources (inclu…☆61Updated 9 months ago
- Ontologies of Linguistic Annotation. Machine-readable tagsets and annotation schemata for more than 100 languages.☆20Updated 2 months ago
- Multilingual Open Text☆25Updated 3 months ago
- This is a new backend implementation of the ANNIS linguistic search and visualization system.☆17Updated this week