Alternatives and similar repositories for chatnoir-resiliparse

Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below

• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆171Updated last year
• allenai / peS2o
  Pretraining Efficiently on S2ORC!
  ☆165Updated 9 months ago
• EleutherAI / stackexchange-dataset
  Python tools for processing the stackexchange data dumps into a text dataset for Language Models
  ☆83Updated last year
• EleutherAI / openwebtext2
  ☆90Updated 3 years ago
• hltcoe / ColBERT-X
  CLIR version of ColBERT
  ☆71Updated last month
• allenai / bff
  ☆39Updated last year
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆104Updated last year
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆85Updated 2 months ago
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆223Updated 8 months ago
• sileod / tasksource
  Datasets collection and preprocessings framework for NLP extreme multitask learning
  ☆185Updated 3 weeks ago
• allenai / catwalk
  This project studies the performance and robustness of language models and task-adaptation methods.
  ☆150Updated last year
• google-deepmind / xtr
  XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
  ☆52Updated last year
• shjwudp / c4-dataset-script
  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…
  ☆129Updated 2 years ago
• illuin-tech / contextual-embeddings
  Model implementation for the contextual embeddings project
  ☆35Updated 2 months ago
• microsoft / MS-MARCO-Web-Search
  A large-scale information-rich web dataset, featuring millions of real clicked query-document labels
  ☆340Updated 7 months ago
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆49Updated last year
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆57Updated last week
• leogao2 / lm_dataformat
  ☆79Updated last year
• castorini / umbrela
  ☆52Updated 2 weeks ago
• lemurproject / ClueWeb22
  ☆16Updated 7 months ago
• huggingface / fineweb-2
  ☆174Updated last month
• allenai / mmda
  multimodal document analysis
  ☆165Updated last year
• huggingface / olm-datasets
  Pipeline for pulling and processing online language model pretraining data from the web
  ☆177Updated 2 years ago
• applicaai / CCpdf
  Index of URLs to pdf files all over the internet and scripts
  ☆24Updated 2 years ago
• cisnlp / GlotLID
  💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
  ☆147Updated 2 months ago
• bigcode-project / bigcode-analysis
  Repository for analysis and experiments in the BigCode project.
  ☆121Updated last year
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆184Updated last week
• Snowflake-Labs / arctic-embed
  ☆76Updated 7 months ago
• mixedbread-ai / baguetter
  Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…
  ☆186Updated 11 months ago
• DunZhang / Stella
  ☆62Updated last year