alexandres / terashuf
terashuf shuffles multi-terabyte text files using limited memory
☆205Updated last year
Related projects ⓘ
Alternatives and complementary repositories for terashuf
- Fast BPE☆656Updated 5 months ago
- An efficient implementation of the popular sequence models for text generation, summarization, and translation tasks. https://arxiv.org/p…☆431Updated 2 years ago
- Corpus preprocessing☆95Updated 8 months ago
- Flexible classic and NeurAl Retrieval Toolkit☆214Updated 4 months ago
- Neural Text Generation with Unlikelihood Training☆310Updated 3 years ago
- Embedding Quantization (Compress Word Embeddings)☆86Updated 5 years ago
- Python port of Moses tokenizer, truecaser and normalizer☆488Updated 5 months ago
- Python library & examples for Masked Language Model Scoring (ACL 2020)☆336Updated last year
- Efficient, check-pointed data loading for deep learning with massive data sets.☆205Updated last year
- PyTorch Language Model for 1-Billion Word (LM1B / GBW) Dataset☆123Updated 5 years ago
- Implementation of a linear-chain CRF in PyTorch☆95Updated 3 years ago
- A tool for holistic analysis of language generations systems☆467Updated 2 years ago
- scripts and configuration files for Edinburgh neural MT submission to WMT 16 shared translation task☆138Updated 4 years ago
- MPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf☆288Updated 3 years ago
- Scripts to preprocess training and test data and to run fast_align and giza☆109Updated 3 years ago
- Neural macine translation soft alignment visualisations for web and command line☆72Updated 3 years ago
- Implementation of unsupervised smoothed inverse frequency (Best Paper, Repl4NLP @ ACL 2018)☆77Updated 5 years ago
- Standalone Neural Ranking Model (SNRM)☆74Updated 5 years ago
- Efficient Markov Chain word alignment☆54Updated 3 years ago
- Byte Pair Encoding for Python!☆223Updated 2 years ago
- This is a repository with the data and code for the ACL 2019 paper "When a Good Translation is Wrong in Context: ..." and the EMNLP 2019 …☆97Updated 4 years ago
- Dataset for the Emerging & Novel Entity NER task (WNUT '17)☆111Updated 2 years ago
- An extension of word2vec to learn phrase embeddings☆73Updated 6 years ago
- CrossWeigh: Training Named Entity Tagger from Imperfect Annotations☆176Updated 3 months ago
- LASER multilingual sentence embeddings as a pip package☆225Updated last year
- We release a dataset based on Wikipedia sentences and the corresponding translations in 6 different languages along with the scores (scal…☆81Updated 3 years ago
- ☆161Updated 4 years ago
- LM Pretraining with PyTorch/TPU☆132Updated 5 years ago
- DeepCT and HDCT uses BERT to generate novel, context-aware bag-of-words term weights for documents and queries.☆313Updated 3 years ago
- ICLR 2018 Quick-Thought vectors☆205Updated 5 years ago