alexandres / terashufLinks
terashuf shuffles multi-terabyte text files using limited memory
☆226Updated 2 years ago
Alternatives and similar repositories for terashuf
Users that are interested in terashuf are comparing it to the libraries listed below
Sorting:
- Byte Pair Encoding for Python!☆232Updated 3 years ago
- An efficient implementation of the popular sequence models for text generation, summarization, and translation tasks. https://arxiv.org/p…☆432Updated 3 years ago
- Implementation of a linear-chain CRF in PyTorch☆98Updated 4 years ago
- Fast BPE☆678Updated last year
- Efficient, check-pointed data loading for deep learning with massive data sets.☆209Updated 2 years ago
- Neural Text Generation with Unlikelihood Training☆310Updated 4 years ago
- This repository contains source code to binarize any real-value word embeddings into binary vectors.☆47Updated 4 years ago
- Flexible classic and NeurAl Retrieval Toolkit☆220Updated 4 months ago
- 📃Language Model based sentences scoring library☆309Updated 3 years ago
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines☆137Updated 2 years ago
- LASER multilingual sentence embeddings as a pip package☆225Updated 2 years ago
- MPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf☆292Updated 4 years ago
- LM Pretraining with PyTorch/TPU☆136Updated 6 years ago
- A python true casing utility that restores case information for texts☆89Updated 2 years ago
- Implementation of unsupervised smoothed inverse frequency (Best Paper, Repl4NLP @ ACL 2018)☆78Updated 6 years ago
- This is a repository with the data and code for the ACL 2019 paper "When a Good Translation is Wrong in Context: ..." and the EMNLP 2019 …☆98Updated 5 years ago
- One million English sentences, each split into two sentences that together preserve the original meaning, extracted from Wikipedia edits.☆123Updated 6 years ago
- Python library & examples for Masked Language Model Scoring (ACL 2020)☆348Updated 2 years ago
- Code and data to support the paper "PAQ 65 Million Probably-Asked Questions andWhat You Can Do With Them"☆207Updated 4 years ago
- Binary Passage Retriever (BPR) - an efficient passage retriever for open-domain question answering☆172Updated 4 years ago
- Neural macine translation soft alignment visualisations for web and command line☆72Updated 4 years ago
- Corpus preprocessing☆99Updated last year
- An official implementation of "BPE-Dropout: Simple and Effective Subword Regularization" algorithm.☆53Updated 4 years ago
- Hyperparameter Search for AllenNLP☆140Updated 7 months ago
- Fast + Non-Autoregressive Grammatical Error Correction using BERT. Code and Pre-trained models for paper "Parallel Iterative Edit Models …☆231Updated 2 years ago
- XtremeDistil framework for distilling/compressing massive multilingual neural network models to tiny and efficient models for AI at scale☆156Updated last year
- PyTorch Language Model for 1-Billion Word (LM1B / GBW) Dataset☆123Updated 6 years ago
- Scripts to preprocess training and test data and to run fast_align and giza☆108Updated 4 years ago
- Robust and Fast tokenizations alignment library for Rust and Python https://tamuhey.github.io/tokenizations/☆193Updated 2 years ago
- ☆325Updated 4 years ago