chatnoir-eu / chatnoir-resiliparseLinks
A robust web archive analytics toolkit
☆114Updated 5 months ago
Alternatives and similar repositories for chatnoir-resiliparse
Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below
Sorting:
- The pipeline for the OSCAR corpus☆171Updated last year
- Pretraining Efficiently on S2ORC!☆165Updated 10 months ago
- Python tools for processing the stackexchange data dumps into a text dataset for Language Models☆83Updated last year
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆186Updated last month
- What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets☆223Updated 9 months ago
- ☆39Updated last year
- This project studies the performance and robustness of language models and task-adaptation methods.☆151Updated last year
- Python API for https://vespa.ai, the open big data serving engine☆137Updated this week
- A large-scale information-rich web dataset, featuring millions of real clicked query-document labels☆340Updated 8 months ago
- Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…☆130Updated 2 years ago
- Model implementation for the contextual embeddings project☆35Updated 2 months ago
- A Python Search Engine for Humans 🥸☆231Updated last year
- ☆62Updated last year
- 💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023☆148Updated 2 months ago
- ☆16Updated 8 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…☆188Updated 11 months ago
- ☆53Updated last month
- Tools for managing datasets for governance and training.☆85Updated 2 weeks ago
- CLIR version of ColBERT☆72Updated 2 months ago
- multimodal document analysis☆164Updated last year
- One-stop shop for running and fine-tuning transformer-based language models for retrieval☆59Updated 2 weeks ago
- ☆183Updated 2 months ago
- [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.☆110Updated last year
- ☆90Updated 3 years ago
- XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval☆53Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆270Updated last year
- [ACL 2025] AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark☆153Updated last month
- Pipeline for pulling and processing online language model pretraining data from the web☆177Updated 2 years ago
- 80x faster and 95% accurate language identification with Fasttext☆162Updated last year