Alternatives and similar repositories for chatnoir-resiliparse

Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below

• allenai / peS2o
  Pretraining Efficiently on S2ORC!
  ☆164Updated 6 months ago
• allenai / catwalk
  This project studies the performance and robustness of language models and task-adaptation methods.
  ☆150Updated 11 months ago
• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆167Updated last year
• EleutherAI / stackexchange-dataset
  Python tools for processing the stackexchange data dumps into a text dataset for Language Models
  ☆82Updated last year
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆101Updated last year
• allenai / bff
  ☆38Updated last year
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆218Updated 6 months ago
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆85Updated 3 months ago
• facebookresearch / SemDeDup
  Code for "SemDeDup", a simple method for identifying and removing semantic duplicates from a dataset (data pairs which are semantically s…
  ☆136Updated last year
• sileod / tasksource
  Datasets collection and preprocessings framework for NLP extreme multitask learning
  ☆180Updated 4 months ago
• r-three / common-pile
  Repo to hold code and track issues for the collection of permissively licensed data
  ☆24Updated last week
• shjwudp / c4-dataset-script
  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…
  ☆124Updated last year
• EleutherAI / openwebtext2
  ☆90Updated 2 years ago
• huggingface / llm-swarm
  Manage scalable open LLM inference endpoints in Slurm clusters
  ☆256Updated 10 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 11 months ago
• chatnoir-eu / web-content-extraction-benchmark
  Web Content Extraction Benchmark
  ☆17Updated 11 months ago
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆123Updated last week
• p-lambda / dsir
  DSIR large-scale data selection framework for language model training
  ☆247Updated last year
• AI21Labs / Parallel-Context-Windows
  ☆106Updated last year
• mixedbread-ai / baguetter
  Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…
  ☆177Updated 8 months ago
• Hannibal046 / nanoColBERT
  Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).
  ☆80Updated last year
• mungg / FABLES
  ☆57Updated 7 months ago
• marzenakrp / nocha
  ☆51Updated 6 months ago
• AlexWan0 / infini-gram
  An unofficial implementation of the Infini-gram model proposed by Liu et al. (2024)
  ☆32Updated 10 months ago
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆53Updated this week
• shayne-longpre / a-pretrainers-guide
  ☆72Updated last year
• mgmalek / efficient_cross_entropy
  ☆106Updated 11 months ago
• lm-sys / llm-decontaminator
  Code for the paper "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples"
  ☆301Updated last year
• dwzhu-pku / LongEmbed
  LongEmbed: Extending Embedding Models for Long Context Retrieval (EMNLP 2024)
  ☆135Updated 6 months ago
• google-deepmind / xtr
  XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
  ☆51Updated 10 months ago