sdtblck / Opensubtitles_dataset
downloads and parses subtitle dataset from opensubtitles.org
☆16Updated 5 months ago
Related projects: ⓘ
- A guide to building language technology in new languages.☆57Updated 2 years ago
- OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.☆45Updated 2 weeks ago
- Complimentary code for our paper Automatic punctuation restoration with BERT models☆48Updated 10 months ago
- The pipeline for the OSCAR corpus☆161Updated 9 months ago
- Tool to fix bitexts and tag near-duplicates for removal☆29Updated last month
- Bicleaner fork that uses neural networks☆37Updated last month
- Reduce the size of pretrained Hugging Face models via vocabulary trimming.☆39Updated last year
- Implementation of "SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages" paper, accepted to E…☆21Updated last year
- XtremeDistil framework for distilling/compressing massive multilingual neural network models to tiny and efficient models for AI at scale☆152Updated 9 months ago
- Tutorial to pretrain & fine-tune a 🤗 Flax T5 model on a TPUv3-8 with GCP☆58Updated 2 years ago
- Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)☆65Updated 4 months ago
- A tiny BERT for low-resource monolingual models☆28Updated 4 months ago
- Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.☆73Updated last week
- A Benchmark Dataset for Understanding Disfluencies in Question Answering☆60Updated 3 years ago
- ☆13Updated 2 years ago
- ☆86Updated 2 years ago
- This repository contains the code for "Generating Datasets with Pretrained Language Models".☆188Updated 3 years ago
- This repository contains data used in the NAACL 2021 Paper - Proteno: Text Normalization with Limited Data for Fast Deployment in Text to…☆41Updated 3 years ago
- 📝An easy-to-use package to restore punctuation of the text.☆107Updated last year
- Library for pruning experts per language pair in NLLB-200☆26Updated last year
- Experiments with generating opensource language model assistants☆97Updated last year
- A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB te…☆247Updated 9 months ago
- A collection of scripts to preprocess ASR datasets and finetune language-specific Wav2Vec2 XLSR models☆31Updated 3 years ago
- fastlangid, the only language identification package that support cantonese (zh-yue), simplified (zh-hans) and traditional chinese (zh-ha…☆38Updated last year
- GlotLID: Language Identification with Support for More Than 2000 Labels -- EMNLP 2023☆85Updated 2 months ago
- A tool that locates, downloads, and extracts machine translation corpora☆145Updated 3 months ago
- An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.☆85Updated 3 years ago
- A python true casing utility that restores case information for texts☆88Updated last year
- xfspell — the Transformer Spell Checker☆186Updated 4 years ago
- Curriculum training☆15Updated last week