Alternatives and similar repositories for Tokenizer:

Users that are interested in Tokenizer are comparing it to the libraries listed below

• thammegowda / mtdata
  A tool that locates, downloads, and extracts machine translation corpora
  ☆149Updated 8 months ago
• bitextor / bicleaner
  Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.
  ☆152Updated 7 months ago
• bitextor / bitextor
  Bitextor generates translation memories from multilingual websites
  ☆293Updated 2 months ago
• marian-nmt / marian-examples
  Examples, tutorials and use cases for Marian, including our WMT-2017/18 baselines.
  ☆76Updated last year
• glample / fastBPE
  Fast BPE
  ☆659Updated 7 months ago
• Helsinki-NLP / OpusFilter
  OpusFilter - Parallel corpus processing toolkit
  ☆104Updated last week
• neulab / awesome-align
  A neural word aligner based on multilingual BERT
  ☆336Updated 2 years ago
• marian-nmt / marian-dev
  Fast Neural Machine Translation in C++ - development repository
  ☆258Updated 3 months ago
• robertostling / eflomal
  Efficient Low-Memory Aligner
  ☆140Updated 2 weeks ago
• thompsonb / vecalign
  Improved Sentence Alignment in Linear Time and Space
  ☆163Updated last year
• cisnlp / simalign
  Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
  ☆354Updated last year
• moses-smt / giza-pp
  GIZA++ is a statistical machine translation toolkit that is used to train IBM Models 1-5 and an HMM word alignment model. This package al…
  ☆264Updated last year
• moses-smt / mgiza
  A word alignment tool based on famous GIZA++, extended to support multi-threading, resume training and incremental training.
  ☆161Updated 3 years ago
• kpu / preprocess
  Corpus preprocessing
  ☆95Updated 10 months ago
• danielvarga / hunalign
  Sentence aligner
  ☆108Updated 3 years ago
• hplt-project / sacremoses
  Python port of Moses tokenizer, truecaser and normalizer
  ☆490Updated 8 months ago
• clab / fast_align
  Simple, fast unsupervised word aligner
  ☆743Updated 2 years ago
• Unbabel / OpenKiwi
  Open-Source Machine Translation Quality Estimation in PyTorch
  ☆229Updated 2 years ago
• neulab / xnmt
  eXtensible Neural Machine Translation
  ☆185Updated 4 years ago
• rsennrich / wmt16-scripts
  scripts and configuration files for Edinburgh neural MT submission to WMT 16 shared translation task
  ☆138Updated 4 years ago
• keisks / jfleg
  JFLEG (JHU FLuency-Extended GUG) corpus for Grammatical Error Correction Evaluation
  ☆112Updated last year
• neulab / compare-mt
  A tool for holistic analysis of language generations systems
  ☆467Updated 2 years ago
• rsennrich / Bleualign
  Machine-Translation-based sentence alignment tool for parallel text
  ☆303Updated 3 years ago
• Hyperparticle / udify
  A single model that parses Universal Dependencies across 75 languages. Given a sentence, jointly predicts part-of-speech tags, morphology…
  ☆221Updated 2 years ago
• lilt / alignment-scripts
  Scripts to preprocess training and test data and to run fast_align and giza
  ☆108Updated 3 years ago
• hainan-xv / zipporah
  ☆42Updated 6 years ago
• SYSTRAN / fuzzy-match
  Library and command line utility to do approximate string matching of a source against a bitext index and get matched source and target.
  ☆47Updated last month
• kanyun-inc / fairseq-gec
  Source code for paper: Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data
  ☆247Updated 4 years ago
• jhclark / tercom
  Translation Error Rate (TER)
  ☆43Updated 6 years ago
• antho-rousseau / XenC
  XenC: open-source data selection tool for NLP
  ☆60Updated 8 years ago