Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• marian-nmt / marian-dev
  Fast Neural Machine Translation in C++ - development repository
  ☆273Updated 7 months ago
• glample / fastBPE
  Fast BPE
  ☆671Updated 11 months ago
• bitextor / bitextor
  Bitextor generates translation memories from multilingual websites
  ☆293Updated 6 months ago
• neulab / awesome-align
  A neural word aligner based on multilingual BERT
  ☆348Updated 3 years ago
• thammegowda / mtdata
  A tool that locates, downloads, and extracts machine translation corpora
  ☆155Updated last week
• bitextor / bicleaner
  Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.
  ☆157Updated 11 months ago
• Hyperparticle / udify
  A single model that parses Universal Dependencies across 75 languages. Given a sentence, jointly predicts part-of-speech tags, morphology…
  ☆222Updated 2 years ago
• cisnlp / simalign
  Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
  ☆361Updated last year
• bojone / labse
  Language-agnostic BERT Sentence Embedding (LaBSE)
  ☆151Updated 4 years ago
• marian-nmt / marian-examples
  Examples, tutorials and use cases for Marian, including our WMT-2017/18 baselines.
  ☆77Updated 2 years ago
• hplt-project / sacremoses
  Python port of Moses tokenizer, truecaser and normalizer
  ☆494Updated last year
• aalto-speech / morfessor
  Morfessor is a tool for unsupervised and semi-supervised morphological segmentation
  ☆193Updated 4 years ago
• kpu / preprocess
  Corpus preprocessing
  ☆97Updated last year
• microsoft / fastseq
  An efficient implementation of the popular sequence models for text generation, summarization, and translation tasks. https://arxiv.org/p…
  ☆433Updated 2 years ago
• Helsinki-NLP / OpusFilter
  OpusFilter - Parallel corpus processing toolkit
  ☆104Updated 2 months ago
• moses-smt / giza-pp
  GIZA++ is a statistical machine translation toolkit that is used to train IBM Models 1-5 and an HMM word alignment model. This package al…
  ☆266Updated 2 years ago
• clab / fast_align
  Simple, fast unsupervised word aligner
  ☆754Updated 2 years ago
• robertostling / eflomal
  Efficient Low-Memory Aligner
  ☆144Updated 4 months ago
• awasthiabhijeet / PIE
  Fast + Non-Autoregressive Grammatical Error Correction using BERT. Code and Pre-trained models for paper "Parallel Iterative Edit Models …
  ☆230Updated 2 years ago
• nusnlp / m2scorer
  MaxMatch (M^2) Scorer - Evaluation program for grammatical error correction systems.
  ☆152Updated 2 years ago
• d99kris / spacy-cpp
  C++ wrapper library for the NLP library spaCy
  ☆103Updated 2 years ago
• UniversalDependencies / UD_English-EWT
  English data
  ☆207Updated this week
• ringgaard / sling
  SLING - A natural language frame semantics parser
  ☆164Updated last week
• nikitakit / self-attentive-parser
  High-accuracy NLP parser with models for 11 languages.
  ☆885Updated 3 years ago
• chrisjbryant / errant
  ERRor ANnotation Toolkit: Automatically extract and classify grammatical errors in parallel original and corrected sentences.
  ☆445Updated last year
• SYSTRAN / fuzzy-match
  Library and command line utility to do approximate string matching of a source against a bitext index and get matched source and target.
  ☆50Updated last month
• google-research-datasets / clang8
  cLang-8 is a dataset for grammatical error correction.
  ☆105Updated 2 years ago
• neulab / compare-mt
  A tool for holistic analysis of language generations systems
  ☆468Updated 3 years ago
• rsennrich / wmt16-scripts
  scripts and configuration files for Edinburgh neural MT submission to WMT 16 shared translation task
  ☆138Updated 4 years ago
• google-research-datasets / word_sense_disambigation_corpora
  SemCor and Masc documents annotated with NOAD word senses.
  ☆183Updated 5 years ago