Alternatives and similar repositories for chatnoir-resiliparse

Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below

• EleutherAI / stackexchange-dataset
  Python tools for processing the stackexchange data dumps into a text dataset for Language Models
  ☆80Updated last year
• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆167Updated last year
• shjwudp / c4-dataset-script
  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…
  ☆126Updated last year
• allenai / catwalk
  This project studies the performance and robustness of language models and task-adaptation methods.
  ☆150Updated last year
• r-three / common-pile
  Code for collecting, processing, and preparing datasets for the Common Pile
  ☆27Updated this week
• lemurproject / ClueWeb22
  ☆16Updated 5 months ago
• allenai / peS2o
  Pretraining Efficiently on S2ORC!
  ☆164Updated 7 months ago
• EleutherAI / openwebtext2
  ☆90Updated 2 years ago
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆221Updated 6 months ago
• allenai / bff
  ☆38Updated last year
• leogao2 / lm_dataformat
  ☆78Updated last year
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆85Updated last week
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆56Updated last week
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆48Updated last year
• google-deepmind / xtr
  XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
  ☆51Updated 11 months ago
• Snowflake-Labs / arctic-embed
  ☆70Updated 5 months ago
• shayne-longpre / a-pretrainers-guide
  ☆72Updated 2 years ago
• marzenakrp / nocha
  ☆51Updated 7 months ago
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆313Updated 2 years ago
• AmenRa / indxr
  A Python utility for indexing file lines. Best demo honourable mention at ECIR 2024.
  ☆22Updated last year
• google-research / t5x_retrieval
  ☆98Updated 2 years ago
• commoncrawl / ia-web-commons
  Web archiving utility library
  ☆11Updated 2 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated last year
• huggingface / llm-swarm
  Manage scalable open LLM inference endpoints in Slurm clusters
  ☆258Updated 10 months ago
• mixedbread-ai / baguetter
  Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…
  ☆179Updated 9 months ago
• hltcoe / ColBERT-X
  CLIR version of ColBERT
  ☆67Updated last month
• huggingface / olm-datasets
  Pipeline for pulling and processing online language model pretraining data from the web
  ☆178Updated last year
• AI21Labs / Parallel-Context-Windows
  ☆105Updated last year
• daniel-furman / sft-demos
  Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
  ☆77Updated 7 months ago
• facebookresearch / SemDeDup
  Code for "SemDeDup", a simple method for identifying and removing semantic duplicates from a dataset (data pairs which are semantically s…
  ☆136Updated last year