Alternatives and similar repositories for fast-mosestokenizer

Users that are interested in fast-mosestokenizer are comparing it to the libraries listed below

• marian-nmt / sotastream
  A library for data streaming and augmentation
  ☆20Updated 5 months ago
• mayhewsw / pytorch-truecaser
  A simple neural truecaser written in pytorch and allennlp.
  ☆33Updated last year
• cambridgeltl / parameter-factorization
  Factorization of the neural parameter space for zero-shot multi-lingual and multi-task transfer
  ☆39Updated 5 years ago
• allenai / tpu_pretrain
  LM Pretraining with PyTorch/TPU
  ☆136Updated 5 years ago
• harvardnlp / hmm-lm
  ☆41Updated 4 years ago
• kpu / preprocess
  Corpus preprocessing
  ☆99Updated last year
• matthewfl / openfst-wrapper
  ☆28Updated 4 years ago
• bitextor / bifixer
  Tool to fix bitexts and tag near-duplicates for removal
  ☆33Updated last month
• oscar-project / goclassy
  An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.
  ☆86Updated 4 years ago
• TimDettmers / transformer-xl
  ☆64Updated 5 years ago
• shijie-wu / neural-transducer
  This repo contains a set of neural transducer, e.g. sequence-to-sequence model, focusing on character-level tasks.
  ☆76Updated 2 years ago
• yanaiela / num_fh
  numeric fused-head identification and resolution
  ☆33Updated 5 years ago
• mayhewsw / multilingual-data-stats
  Statistics on multilingual datasets
  ☆17Updated 3 years ago
• trevorcohn / mantis
  Deep learning model of machine translation using attentional and structural biases
  ☆13Updated 8 years ago
• nyu-mll / pretraining-learning-curves
  The repository for the paper "When Do You Need Billions of Words of Pretraining Data?"
  ☆21Updated 4 years ago
• cindyxinyiwang / SDE
  Source code for the paper "Multilingual Neural Machine Translation with Soft Decoupled Encoding"
  ☆29Updated 4 years ago
• sgraaf / Replicate-Toronto-BookCorpus
  This repository contains code to replicate the no-longer publicly available Toronto BookCorpus dataset
  ☆49Updated 3 years ago
• noe / fairseq-tensorboard
  Small utility to monitor fairseq training in tensorboard
  ☆21Updated 6 years ago
• jungokasai / deep-shallow
  ☆44Updated 5 years ago
• cccwam / rc2020_electra
  ML Reproducibility Challenge 2020: Electra reimplementation using PyTorch and Transformers
  ☆12Updated 4 years ago
• MultiPath / Efficient-Neural-Machine-Translation
  PhD thesis (updating) of Jiatao Gu from HKU
  ☆19Updated 7 years ago
• nikitakit / sabertooth
  Standalone pre-training recipe with JAX+Flax
  ☆32Updated 2 years ago
• zomux / lanmt
  LaNMT: Latent-variable Non-autoregressive Neural Machine Translation with Deterministic Inference
  ☆79Updated 4 years ago
• tca19 / near-lossless-binarization
  This repository contains source code to binarize any real-value word embeddings into binary vectors.
  ☆47Updated 4 years ago
• spyysalo / wiki-bert-pipeline
  Generate BERT vocabularies and pretraining examples from Wikipedias
  ☆17Updated 5 years ago
• jekbradbury / revtok
  Reversible tokenization in Python.
  ☆60Updated 7 years ago
• ofirpress / shortformer
  Code for the Shortformer model, from the ACL 2021 paper by Ofir Press, Noah A. Smith and Mike Lewis.
  ☆147Updated 4 years ago
• VProv / BPE-Dropout
  An official implementation of "BPE-Dropout: Simple and Effective Subword Regularization" algorithm.
  ☆52Updated 4 years ago
• claravania / subword-lstm-lm
  LSTM Language Model with Subword Units Input Representations
  ☆42Updated 4 years ago
• jwieting / paraphrastic-representations-at-scale
  ☆75Updated 4 years ago