chatnoir-eu / chatnoir-resiliparseLinks
A robust web archive analytics toolkit
☆111Updated 3 months ago
Alternatives and similar repositories for chatnoir-resiliparse
Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below
Sorting:
- Pretraining Efficiently on S2ORC!☆164Updated 8 months ago
- Python tools for processing the stackexchange data dumps into a text dataset for Language Models☆81Updated last year
- This project studies the performance and robustness of language models and task-adaptation methods.☆149Updated last year
- The pipeline for the OSCAR corpus☆169Updated last year
- Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…☆127Updated 2 years ago
- ☆38Updated last year
- What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets☆221Updated 7 months ago
- ☆16Updated 6 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆101Updated last year
- multimodal document analysis☆165Updated last year
- Web Content Extraction Benchmark☆17Updated last year
- Web archiving utility library☆11Updated 3 months ago
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆48Updated last year
- Python API for https://vespa.ai, the open big data serving engine☆127Updated last week
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆184Updated 5 months ago
- ☆126Updated last week
- Python package for serving a local search engine. One command to download and serve a datastore---that's it 😎.☆21Updated 2 weeks ago
- Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.☆93Updated 2 years ago
- Index of URLs to pdf files all over the internet and scripts☆24Updated 2 years ago
- Scalable training for dense retrieval models.☆298Updated 2 weeks ago
- One-stop shop for running and fine-tuning transformer-based language models for retrieval☆56Updated last week
- ☆72Updated 2 years ago
- Tools for managing datasets for governance and training.☆85Updated last month
- Code used for the creation of OBELICS, an open, massive and curated collection of interleaved image-text web documents, containing 141M d…☆202Updated 9 months ago
- Manage scalable open LLM inference endpoints in Slurm clusters☆261Updated 11 months ago
- An Experiment on Dynamic NTK Scaling RoPE☆64Updated last year
- Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…☆181Updated 9 months ago
- ☆74Updated 6 months ago
- [ICLR 2023] Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners☆116Updated 9 months ago
- Our open source implementation of MiniLMv2 (https://aclanthology.org/2021.findings-acl.188)☆61Updated 2 years ago