Alternatives and similar repositories for ia-web-commons

Users that are interested in ia-web-commons are comparing it to the libraries listed below

• allenai / bff
  ☆38Updated last year
• lemurproject / ClueWeb22
  ☆16Updated 5 months ago
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆101Updated last year
• shayne-longpre / a-pretrainers-guide
  ☆72Updated 2 years ago
• sebastian-hofstaetter / colberter
  ☆47Updated 3 years ago
• huggingface / olm-datasets
  Pipeline for pulling and processing online language model pretraining data from the web
  ☆178Updated last year
• bigscience-workshop / lm-evaluation-harness
  A framework for few-shot evaluation of autoregressive language models.
  ☆103Updated 2 years ago
• thakur-nandan / income
  INCOME: An Easy Repository for Training and Evaluation of Index Compression Methods in Dense Retrieval. Includes BPR and JPQ.
  ☆24Updated last year
• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆167Updated last year
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆221Updated 6 months ago
• allenai / peS2o
  Pretraining Efficiently on S2ORC!
  ☆164Updated 7 months ago
• google-research / t5x_retrieval
  ☆98Updated 2 years ago
• tau-nlp / scrolls
  The official code of EMNLP 2022, "SCROLLS: Standardized CompaRison Over Long Language Sequences".
  ☆69Updated last year
• castorini / dhr
  Dense hybrid representations for text retrieval
  ☆62Updated 2 years ago
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆85Updated last week
• thongnt99 / learned-sparse-retrieval
  Unified Learned Sparse Retrieval Framework
  ☆64Updated last year
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated last year
• huggingface / olm-training
  Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.
  ☆93Updated 2 years ago
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆128Updated 4 months ago
• r-three / common-pile
  Code for collecting, processing, and preparing datasets for the Common Pile
  ☆27Updated this week
• chaitanyamalaviya / ExpertQA
  [Data + code] ExpertQA : Expert-Curated Questions and Attributed Answers
  ☆128Updated last year
• huggingface / that_is_good_data
  ☆65Updated last year
• castorini / mr.tydi
  Mr. TyDi is a multi-lingual benchmark dataset built on TyDi, covering eleven typologically diverse languages.
  ☆76Updated 3 years ago
• studio-ousia / bpr
  Binary Passage Retriever (BPR) - an efficient passage retriever for open-domain question answering
  ☆170Updated 4 years ago
• allenai / catwalk
  This project studies the performance and robustness of language models and task-adaptation methods.
  ☆150Updated last year
• terrierteam / ir_measures
  provides a common interface to many IR measure tools
  ☆84Updated 3 weeks ago
• thakur-nandan / sprint
  SPRINT Toolkit helps you evaluate diverse neural sparse models easily using a single click on any IR dataset.
  ☆45Updated last year
• EleutherAI / improved-t5
  Experiments for efforts to train a new and improved t5
  ☆77Updated last year
• castorini / chatty-goose
  A Python framework for conversational search
  ☆40Updated 3 years ago
• kernelmachine / silo-lm
  SILO Language Models code repository
  ☆81Updated last year