hplt-project / sacremoses
Python port of Moses tokenizer, truecaser and normalizer
☆492Updated 9 months ago
Alternatives and similar repositories for sacremoses:
Users that are interested in sacremoses are comparing it to the libraries listed below
- Simple, fast unsupervised word aligner☆749Updated 2 years ago
- A tool for holistic analysis of language generations systems☆467Updated 3 years ago
- Fast BPE☆666Updated 9 months ago
- Unsupervised Statistical Machine Translation☆229Updated 4 years ago
- Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)☆361Updated last year
- A neural word aligner based on multilingual BERT☆342Updated 3 years ago
- Open-Source Machine Translation Quality Estimation in PyTorch☆230Updated 2 years ago
- Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.☆155Updated 9 months ago
- A tool that locates, downloads, and extracts machine translation corpora☆154Updated this week
- This dataset contains 108,463 human-labeled and 656k noisily labeled pairs that feature the importance of modeling structure, context, an…☆557Updated 3 years ago
- Builds wordpiece(subword) vocabulary compatible for Google Research's BERT☆228Updated 4 years ago
- A CoNLL-U parser that takes a CoNLL-U formatted string and turns it into a nested python dictionary.☆314Updated last month
- Open-Source Neural Machine Translation in Tensorflow☆797Updated 2 years ago
- A Python wrapper for the ROUGE summarization evaluation package☆251Updated 4 years ago
- A framework to learn cross-lingual word embedding mappings☆647Updated last year
- Improved Sentence Alignment in Linear Time and Space☆169Updated 2 years ago
- Bitextor generates translation memories from multilingual websites☆291Updated 4 months ago
- Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE)☆1,205Updated 5 months ago
- ERRor ANnotation Toolkit: Automatically extract and classify grammatical errors in parallel original and corrected sentences.☆443Updated 11 months ago
- A full Python Implementation of the ROUGE Metric (not a wrapper)☆687Updated 4 months ago
- Deep neural models for core NLP tasks (Pytorch version)☆440Updated 3 years ago
- Text tokenization and sentence segmentation (segtok v2)☆202Updated 3 years ago
- Resources for the NAACL 2018 paper "A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents"☆365Updated last year
- LASER multilingual sentence embeddings as a pip package☆224Updated last year
- High-accuracy NLP parser with models for 11 languages.☆880Updated 3 years ago
- New dataset☆303Updated 3 years ago
- GIZA++ is a statistical machine translation toolkit that is used to train IBM Models 1-5 and an HMM word alignment model. This package al…☆265Updated last year
- Efficient Low-Memory Aligner☆142Updated 2 months ago
- Evaluating Cross-lingual Sentence Representations☆450Updated 3 years ago
- A sentence segmenter that actually works!☆305Updated 4 years ago