bigscience-workshop / data_toolingLinks
Tools for managing datasets for governance and training.
☆86Updated 2 weeks ago
Alternatives and similar repositories for data_tooling
Users that are interested in data_tooling are comparing it to the libraries listed below
Sorting:
- Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.☆96Updated 2 years ago
- Pipeline for pulling and processing online language model pretraining data from the web☆177Updated 2 years ago
- ☆101Updated 2 years ago
- ☆72Updated 2 years ago
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆188Updated 3 months ago
- The pipeline for the OSCAR corpus☆173Updated last year
- BLOOM+1: Adapting BLOOM model to support a new unseen language☆74Updated last year
- ☆65Updated 2 years ago
- Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.☆85Updated last year
- A Multilingual Replicable Instruction-Following Model☆95Updated 2 years ago
- ☆79Updated last year
- Pretraining Efficiently on S2ORC!☆170Updated last year
- XtremeDistil framework for distilling/compressing massive multilingual neural network models to tiny and efficient models for AI at scale☆156Updated last year
- ☆184Updated 2 years ago
- Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023☆106Updated last year
- ☆91Updated 3 years ago
- A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.☆61Updated last year
- Dataset from the paper "Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering" (COLING 2022)☆114Updated 3 years ago
- Using business-level retrieval system (BM25) with Python in just a few lines.☆31Updated 2 years ago
- This project studies the performance and robustness of language models and task-adaptation methods.☆154Updated last year
- Open source library for few shot NLP☆78Updated 2 years ago
- ☆27Updated 8 months ago
- Vocabulary Trimming (VT) is a model compression technique, which reduces a multilingual LM vocabulary to a target language by deleting ir…☆56Updated last year
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- ☆86Updated 6 months ago
- Mr. TyDi is a multi-lingual benchmark dataset built on TyDi, covering eleven typologically diverse languages.☆79Updated 3 years ago
- ☆29Updated last year
- [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"☆34Updated 4 months ago
- [TMLR'23] Contrastive Search Is What You Need For Neural Text Generation☆121Updated 2 years ago
- Apps built using Inspired Cognition's Critique.☆58Updated 2 years ago