Helsinki-NLP / OPUS-MT-train

Related projects ⓘ

Alternatives and complementary repositories for OPUS-MT-train

• Helsinki-NLP / Opus-MT
  Open neural machine translation models and web services
  ☆623Updated last month
• facebookresearch / stopes
  A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB te…
  ☆251Updated last month
• bitextor / bicleaner
  Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.
  ☆150Updated 5 months ago
• google-research-datasets / C4_200M-synthetic-dataset-for-grammatical-error-correction
  This dataset contains synthetic training data for grammatical error correction. The corpus is generated by corrupting clean sentences fro…
  ☆157Updated last month
• Ki6an / fastT5
  ⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.
  ☆565Updated last year
• neulab / awesome-align
  A neural word aligner based on multilingual BERT
  ☆328Updated 2 years ago
• cisnlp / simalign
  Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
  ☆351Updated last year
• bfsujason / bertalign
  Multilingual sentence alignment using sentence embeddings
  ☆101Updated 2 weeks ago
• eole-nlp / eole
  Open language modeling toolkit based on PyTorch
  ☆61Updated this week
• google-research / byt5
  ☆487Updated 9 months ago
• facebookresearch / cc_net
  Tools to download and cleanup Common Crawl data
  ☆971Updated last year
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆230Updated 2 years ago
• facebookresearch / flores
  Facebook Low Resource (FLoRes) MT Benchmark
  ☆704Updated last year
• google-research / multilingual-t5
  ☆1,252Updated last year
• ZNLP / BigTranslate
  BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages
  ☆219Updated last year
• abelriboulot / onnxt5
  Summarization, translation, sentiment-analysis, text-generation and more at blazing speed using a T5 version implemented in ONNX.
  ☆252Updated 2 years ago
• FerdinandZhong / punctuator
  A small seq2seq punctuator tool based on DistilBERT
  ☆50Updated 2 months ago
• marian-nmt / marian
  Fast Neural Machine Translation in C++
  ☆1,254Updated last year
• microsoft / MPNet
  MPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf
  ☆288Updated 3 years ago
• microsoft / fastformers
  FastFormers - highly efficient transformer models for NLU
  ☆701Updated 10 months ago
• Helsinki-NLP / OpusFilter
  OpusFilter - Parallel corpus processing toolkit
  ☆102Updated 3 months ago
• yannvgn / laserembeddings
  LASER multilingual sentence embeddings as a pip package
  ☆225Updated last year
• mjpost / sacrebleu
  Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
  ☆1,068Updated 3 months ago
• thompsonb / vecalign
  Improved Sentence Alignment in Linear Time and Space
  ☆163Updated last year
• microsoft / fastseq
  An efficient implementation of the popular sequence models for text generation, summarization, and translation tasks. https://arxiv.org/p…
  ☆433Updated 2 years ago
• thammegowda / mtdata
  A tool that locates, downloads, and extracts machine translation corpora
  ☆147Updated 5 months ago
• google-research-datasets / clang8
  cLang-8 is a dataset for grammatical error correction.
  ☆103Updated 2 years ago
• pluiez / NLLB-inference
  ☆56Updated 2 years ago
• OpenNMT / Tokenizer
  Fast and customizable text tokenization library with BPE and SentencePiece support
  ☆284Updated 2 months ago
• hplt-project / sacremoses
  Python port of Moses tokenizer, truecaser and normalizer
  ☆488Updated 5 months ago