mingruimingrui / fast-mosestokenizerLinks
c++ mosestokenizer
☆18Updated last year
Alternatives and similar repositories for fast-mosestokenizer
Users that are interested in fast-mosestokenizer are comparing it to the libraries listed below
Sorting:
- A library for data streaming and augmentation☆20Updated 3 months ago
- A simple neural truecaser written in pytorch and allennlp.☆33Updated last year
- Tool to fix bitexts and tag near-duplicates for removal☆31Updated 6 months ago
- Corpus preprocessing☆98Updated last year
- Factorization of the neural parameter space for zero-shot multi-lingual and multi-task transfer☆39Updated 4 years ago
- LM Pretraining with PyTorch/TPU☆135Updated 5 years ago
- ☆28Updated 4 years ago
- This repo contains code to reproduce some of the results presented in the paper "SentenceMIM: A Latent Variable Language Model"☆28Updated 3 years ago
- The repository for the paper "When Do You Need Billions of Words of Pretraining Data?"☆21Updated 4 years ago
- Sampling-Based Minimum Bayes-Risk Decoding for Neural Machine Translation☆16Updated 2 years ago
- Source code for the paper "Multilingual Neural Machine Translation with Soft Decoupled Encoding"☆29Updated 4 years ago
- ☆41Updated 4 years ago
- This repo contains a set of neural transducer, e.g. sequence-to-sequence model, focusing on character-level tasks.☆76Updated last year
- ☆44Updated 4 years ago
- ☆22Updated 5 years ago
- As good as new. How to successfully recycle English GPT-2 to make models for other languages (ACL Findings 2021)☆48Updated 4 years ago
- Post-editing Datasets by Rakuten (PEDRa)☆14Updated 4 years ago
- LaNMT: Latent-variable Non-autoregressive Neural Machine Translation with Deterministic Inference☆80Updated 4 years ago
- ☆46Updated 3 years ago
- This repository contains code to replicate the no-longer publicly available Toronto BookCorpus dataset☆49Updated 3 years ago
- A Benchmark Dataset for Understanding Disfluencies in Question Answering☆63Updated 4 years ago
- ☆64Updated 5 years ago
- Implementation of the GBST block from the Charformer paper, in Pytorch☆118Updated 4 years ago
- An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.☆86Updated 4 years ago
- Implementation of the paper 'Sentence Bottleneck Autoencoders from Transformer Language Models'☆17Updated 3 years ago
- Small utility to monitor fairseq training in tensorboard☆21Updated 6 years ago
- This repository contains the code for running the character-level Sandwich Transformers from our ACL 2020 paper on Improving Transformer …☆55Updated 4 years ago
- Library for experimenting with state-of-the-art evaluation metrics like UScore