wikimedia / sentencex
A sentence segmentation library with wide language support optimized for speed and utility.
☆61Updated 7 months ago
Alternatives and similar repositories for sentencex:
Users that are interested in sentencex are comparing it to the libraries listed below
- Seed Machine Translation Data☆31Updated 5 months ago
- Faster, modernized fork of the language identification tool langid.py☆55Updated 5 months ago
- Targetted language identifier, based on FastText and Hunspell.☆34Updated 2 months ago
- Searching in-memory corpus with Corpus Query Language (CQL)☆19Updated 4 months ago
- OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.☆51Updated 3 months ago
- The Open Parallel Corpus☆70Updated last month
- Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)☆74Updated 3 weeks ago
- Extracts plain text, language identification and more metadata from WARC records☆21Updated last month
- 💥 Use Hugging Face text and token classification pipelines directly in spaCy☆63Updated last year
- Library for fast text representation and classification.☆28Updated last year
- Python Finite-State Toolkit☆54Updated last month
- Character-level conversion between Hebrew text and Latin transliteration using deep learning - a demonstration of seq2seq training.☆13Updated last year
- Tool to fix bitexts and tag near-duplicates for removal☆30Updated 2 months ago
- Multilingual syllable annotation pipeline component for spacy☆39Updated 2 years ago
- MAMMOTH: MAssively Multilingual Modular Open Translation @ Helsinki☆23Updated 2 months ago
- Efficient teacher-student models and scripts to make them☆50Updated last year
- FoLiA Linguistic Annotation Tool -- Flat is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.g…☆112Updated 3 months ago
- Implementation of "SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages" paper, accepted to E…☆25Updated 2 years ago
- Compute PageRank on >3 billion Wikipedia links on off-the-shelf hardware.☆58Updated 5 months ago
- Measure the readability of a given text using surface characteristics☆78Updated 2 months ago
- This packages up data for the Open Multilingual Wordnet☆48Updated this week
- Code for SaGe subword tokenizer (EACL 2023)☆24Updated 4 months ago
- Multilingual sentence alignment using sentence embeddings☆116Updated 5 months ago
- An easy-to-use library to linguistically compare one sentence and its words to another, in the same language or a different one. For inst…☆22Updated 3 years ago
- Bilingual term extractor☆53Updated last year
- NTREX -- News Test References for MT Evaluation☆83Updated 10 months ago
- Bilingual sentence similarity classifier using Tensorflow☆21Updated 5 years ago