malteos / llm-datasetsLinks
A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
☆61Updated last year
Alternatives and similar repositories for llm-datasets
Users that are interested in llm-datasets are comparing it to the libraries listed below
Sorting:
- [Data + code] ExpertQA : Expert-Curated Questions and Attributed Answers☆135Updated last year
- [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.☆110Updated last year
- Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning☆30Updated 2 years ago
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆188Updated 3 months ago
- Tools for managing datasets for governance and training.☆86Updated 2 weeks ago
- ☆65Updated 2 years ago
- Retrieval-Augmented Generation battle!☆59Updated 2 months ago
- BLOOM+1: Adapting BLOOM model to support a new unseen language☆74Updated last year
- Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023☆106Updated last year
- Fine-tune ModernBERT on a large Dataset with Custom Tokenizer Training☆73Updated last week
- Starbucks: Improved Training for 2D Matryoshka Embeddings☆22Updated 3 months ago
- One-stop shop for running and fine-tuning transformer-based language models for retrieval☆59Updated this week
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆49Updated last year
- 💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023☆163Updated 4 months ago
- The official code repo for "Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations".☆83Updated last year
- A Human-LLM Collaborative Dataset for Generative Information-seeking with Attribution☆35Updated 2 years ago
- Pretraining Efficiently on S2ORC!☆170Updated last year
- ☆27Updated 8 months ago
- Repository for "Attribute First, then Generate: Locally-attributable Grounded Text Generation", ACL 2024☆29Updated 10 months ago
- Semantically Structured Sentence Embeddings☆67Updated last year
- ☆86Updated 6 months ago
- SPRINT Toolkit helps you evaluate diverse neural sparse models easily using a single click on any IR dataset.☆47Updated 2 years ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆66Updated 3 weeks ago
- GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embeddings☆44Updated last year
- FastFit ⚡ When LLMs are Unfit Use FastFit ⚡ Fast and Effective Text Classification with Many Classes☆212Updated last month
- Do Multilingual Language Models Think Better in English?☆42Updated 2 years ago
- SQuARE: Software for question answering research.☆75Updated last year
- The official repository for Efficient Long-Text Understanding Using Short-Text Models (Ivgi et al., 2022) paper☆70Updated 2 years ago
- The corresponding code for our paper: "Exploring the Challenges of Open Domain Multi-Document Summarization". Do not hesitate to open an …☆32Updated 2 years ago
- PropSegmEnt is an annotated dataset for segmenting English text into propositions, and recognizing proposition-level entailment relations…☆21Updated 2 years ago