MiniXC / opensubtitles-dataloader
Loads OpenSubtitles v2018 dataset without having to load everything into memory at once. Works well with pytorch.
☆13Updated 4 years ago
Alternatives and similar repositories for opensubtitles-dataloader:
Users that are interested in opensubtitles-dataloader are comparing it to the libraries listed below
- Easily deploy a state-of-the-art language model from HuggingFace's Transformers☆12Updated 5 years ago
- MILES is a multilingual text simplifier inspired by LSBert - A BERT-based lexical simplification approach proposed in 2018. Unlike LSBert…☆49Updated 3 years ago
- Our open source implementation of MiniLMv2 (https://aclanthology.org/2021.findings-acl.188)☆61Updated last year
- Code for SaGe subword tokenizer (EACL 2023)☆24Updated 4 months ago
- Library for fast text representation and classification.☆28Updated last year
- LTG-Bert☆31Updated last year
- A guide to building language technology in new languages.☆58Updated 3 years ago
- Common crawl pretrained sentencepiece tokenizers for English and Japanese for various vocabulary sizes. Also development environment for …☆10Updated 3 years ago
- URL downloader supporting checkpointing and continuous checksumming.☆19Updated last year
- Dialog Flow Engine (DFE) allows you to write conversational services. The service is written by defining a special dialog graph that desc…☆18Updated 2 years ago
- Generate a SQLite database from Wikipedia & Wikidata dumps.☆33Updated last year
- Extremely easy to use sequence to sequence library with attention, for text to text conversion tasks.☆39Updated 4 years ago
- Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer" https://arxiv.org/pdf/2112.14569.pdf☆20Updated 3 years ago
- Tutorial to pretrain & fine-tune a 🤗 Flax T5 model on a TPUv3-8 with GCP☆58Updated 2 years ago
- Documentation effort for the BookCorpus dataset☆34Updated 3 years ago
- Execute arbitrary SQL queries on 🤗 Datasets☆32Updated last year
- Experiments with Hugging Face 🔬 🤗☆44Updated 7 months ago
- This repository contains code to replicate the no-longer publicly available Toronto BookCorpus dataset☆49Updated 2 years ago
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆26Updated 11 months ago
- HomebrewNLP in JAX flavour for maintable TPU-Training☆49Updated last year
- negate_sentence(A Python module that doesn't negate sentences.)☆30Updated 5 months ago
- Code for the paper-"Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm" (https://arxiv.org/abs/2007.14966).☆58Updated 3 years ago
- bin files☆13Updated 2 months ago
- This is a prototype of a multi-lingual suite for named-entity recognition in Python.☆21Updated 11 months ago
- A small library with distillation, quantization and pruning pipelines☆26Updated 3 years ago
- spaCy match and replace, maintaining conjugation☆35Updated 2 years ago
- Efficient teacher-student models and scripts to make them☆50Updated last year
- A slightly opinionated iPython profile for interactive development☆23Updated 3 years ago
- A simple neural truecaser written in pytorch and allennlp.☆33Updated 9 months ago
- Question Generation - Question Answering for Automatic Flashcards☆64Updated 3 years ago