mingruimingrui / fast-mosestokenizerLinks
c++ mosestokenizer
☆18Updated last year
Alternatives and similar repositories for fast-mosestokenizer
Users that are interested in fast-mosestokenizer are comparing it to the libraries listed below
Sorting:
- A library for data streaming and augmentation☆20Updated 3 months ago
- A simple neural truecaser written in pytorch and allennlp.☆33Updated last year
- LM Pretraining with PyTorch/TPU☆135Updated 5 years ago
- Statistics on multilingual datasets☆17Updated 3 years ago
- Tool to fix bitexts and tag near-duplicates for removal☆31Updated 6 months ago
- Corpus preprocessing☆98Updated last year
- Factorization of the neural parameter space for zero-shot multi-lingual and multi-task transfer☆39Updated 4 years ago
- Source code for the paper "Multilingual Neural Machine Translation with Soft Decoupled Encoding"☆29Updated 4 years ago
- This repository contains code to replicate the no-longer publicly available Toronto BookCorpus dataset☆49Updated 3 years ago
- This repo contains a set of neural transducer, e.g. sequence-to-sequence model, focusing on character-level tasks.☆76Updated last year
- An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.☆86Updated 4 years ago
- Implementation of the GBST block from the Charformer paper, in Pytorch☆118Updated 4 years ago
- ☆28Updated 4 years ago
- Small utility to monitor fairseq training in tensorboard☆21Updated 6 years ago
- LaNMT: Latent-variable Non-autoregressive Neural Machine Translation with Deterministic Inference☆80Updated 3 years ago
- The repository for the paper "When Do You Need Billions of Words of Pretraining Data?"☆21Updated 4 years ago
- This repository contains the code for running the character-level Sandwich Transformers from our ACL 2020 paper on Improving Transformer …☆55Updated 4 years ago
- Code for the Shortformer model, from the ACL 2021 paper by Ofir Press, Noah A. Smith and Mike Lewis.☆147Updated 4 years ago
- ☆44Updated 4 years ago
- Standalone pre-training recipe with JAX+Flax☆32Updated 2 years ago
- This repo is containing notes and implementations for cherry-picked publications of my particular interest☆12Updated 5 years ago
- Implementation of the paper 'Sentence Bottleneck Autoencoders from Transformer Language Models'☆17Updated 3 years ago
- ☆64Updated 5 years ago
- Spoken Language Translation System☆13Updated 6 years ago
- Multilingual Open Text☆25Updated 3 months ago
- ☆22Updated 5 years ago
- The 14th Machine Translation Marathon 2019 in Edinburgh☆13Updated 2 years ago
- Reversible tokenization in Python.☆60Updated 6 years ago
- Code and datasets of "Multilingual Extractive Reading Comprehension by Runtime Machine Translation"☆40Updated 6 years ago
- Human evaluation results and translation output for the Translator Human Parity Data release☆37Updated 7 years ago