Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• marian-nmt / marian-dev
  Fast Neural Machine Translation in C++ - development repository
  ☆277Updated 2 months ago
• bitextor / bitextor
  Bitextor generates translation memories from multilingual websites
  ☆295Updated 10 months ago
• bitextor / bicleaner
  Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.
  ☆157Updated last year
• glample / fastBPE
  Fast BPE
  ☆676Updated last year
• thammegowda / mtdata
  A tool that locates, downloads, and extracts machine translation corpora
  ☆156Updated 3 months ago
• Helsinki-NLP / OpusFilter
  OpusFilter - Parallel corpus processing toolkit
  ☆109Updated last month
• marian-nmt / marian-examples
  Examples, tutorials and use cases for Marian, including our WMT-2017/18 baselines.
  ☆77Updated 2 years ago
• clab / fast_align
  Simple, fast unsupervised word aligner
  ☆757Updated 3 years ago
• thompsonb / vecalign
  Improved Sentence Alignment in Linear Time and Space
  ☆181Updated 2 years ago
• danielvarga / hunalign
  Sentence aligner
  ☆117Updated 4 years ago
• Unbabel / OpenKiwi
  Open-Source Machine Translation Quality Estimation in PyTorch
  ☆232Updated 3 years ago
• robertostling / eflomal
  Efficient Low-Memory Aligner
  ☆146Updated 8 months ago
• cisnlp / simalign
  Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
  ☆376Updated last year
• kpu / preprocess
  Corpus preprocessing
  ☆98Updated last year
• hplt-project / sacremoses
  Python port of Moses tokenizer, truecaser and normalizer
  ☆495Updated last year
• moses-smt / giza-pp
  GIZA++ is a statistical machine translation toolkit that is used to train IBM Models 1-5 and an HMM word alignment model. This package al…
  ☆269Updated 2 years ago
• neulab / awesome-align
  A neural word aligner based on multilingual BERT
  ☆356Updated 3 years ago
• chrisjbryant / errant
  ERRor ANnotation Toolkit: Automatically extract and classify grammatical errors in parallel original and corrected sentences.
  ☆450Updated last year
• snukky / wikiedits
  Automatic extraction of edited sentences from text edition histories.
  ☆83Updated 3 years ago
• moses-smt / mgiza
  A word alignment tool based on famous GIZA++, extended to support multi-threading, resume training and incremental training.
  ☆165Updated 4 years ago
• rsennrich / Bleualign
  Machine-Translation-based sentence alignment tool for parallel text
  ☆312Updated 4 years ago
• SYSTRAN / fuzzy-match
  Library and command line utility to do approximate string matching of a source against a bitext index and get matched source and target.
  ☆52Updated 4 months ago
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆254Updated 2 years ago
• facebookresearch / stopes
  A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB te…
  ☆282Updated 7 months ago
• awasthiabhijeet / PIE
  Fast + Non-Autoregressive Grammatical Error Correction using BERT. Code and Pre-trained models for paper "Parallel Iterative Edit Models …
  ☆230Updated 2 years ago
• chrisjbryant / lmgec-lite
  A language model-based approach to Grammatical Error Correction for English that uses minimal annotated data.
  ☆48Updated 6 years ago
• bojone / labse
  Language-agnostic BERT Sentence Embedding (LaBSE)
  ☆153Updated 5 years ago
• OpenNMT / nmt-wizard-docker
  Dockerized NMT frameworks for nmt-wizard
  ☆39Updated 2 years ago
• bytedance / neurst
  Neural end-to-end Speech Translation Toolkit
  ☆309Updated 3 years ago
• NiuTrans / NiuTrans.NMT
  A Fast Neural Machine Translation System developed in C++.
  ☆143Updated last year