wikimedia / sentencex
A sentence segmentation library with wide language support optimized for speed and utility.
☆55Updated 4 months ago
Alternatives and similar repositories for sentencex:
Users that are interested in sentencex are comparing it to the libraries listed below
- Tool to fix bitexts and tag near-duplicates for removal☆29Updated 5 months ago
- Multilingual syllable annotation pipeline component for spacy☆39Updated last year
- Seed Machine Translation Data☆30Updated 2 months ago
- Faster, modernized fork of the language identification tool langid.py☆50Updated 2 months ago
- OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.☆50Updated 2 weeks ago
- Extracts plain text, language identification and more metadata from WARC records☆20Updated 5 months ago
- Transform TMX to text☆28Updated 2 years ago
- Cython wrapper on Hunspell Dictionary☆66Updated 7 months ago
- Python module that identifies Chinese text as being Simplified or Traditional☆87Updated 2 months ago
- Simple multilingual lemmatizer for Python, especially useful for speed and efficiency☆151Updated 2 months ago
- 💥 Use Hugging Face text and token classification pipelines directly in spaCy☆63Updated 10 months ago
- ☆25Updated last year
- Targetted language identifier, based on FastText and Hunspell.☆33Updated 3 months ago
- multimodal document analysis☆162Updated 7 months ago
- The Open Parallel Corpus☆62Updated this week
- ☆68Updated 5 months ago
- Multilingual sentence alignment using sentence embeddings☆106Updated 2 months ago
- Bilingual term extractor☆53Updated last year
- Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)☆69Updated 9 months ago
- Searching in-memory corpus with Corpus Query Language (CQL)☆19Updated last month
- NTREX -- News Test References for MT Evaluation☆80Updated 7 months ago
- An easy-to-use library to linguistically compare one sentence and its words to another, in the same language or a different one. For inst…☆22Updated 3 years ago
- Neural models for detecting and masking personal information from texts☆15Updated 2 years ago
- A python module for word inflections designed for use with spaCy.☆92Updated 4 years ago
- Put together a multilingual corpus from a variety of sources. Used for wordfreq and word embeddings.☆51Updated 3 years ago
- 80x faster and 95% accurate language identification with Fasttext☆145Updated last year
- Sentence aligner☆108Updated 3 years ago
- 🧪 Cutting-edge experimental spaCy components and features☆96Updated 9 months ago
- Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.☆152Updated 7 months ago
- These are lists for a variety of languages containing words that are distinctive to each language.☆35Updated 2 years ago