chatnoir-eu / chatnoir-resiliparseLinks
A robust web archive analytics toolkit
☆117Updated 6 months ago
Alternatives and similar repositories for chatnoir-resiliparse
Users that are interested in chatnoir-resiliparse are comparing it to the libraries listed below
Sorting:
- Pretraining Efficiently on S2ORC!☆170Updated 11 months ago
- The pipeline for the OSCAR corpus☆172Updated last year
- Python tools for processing the stackexchange data dumps into a text dataset for Language Models☆83Updated last year
- This project studies the performance and robustness of language models and task-adaptation methods.☆153Updated last year
- ☆39Updated last year
- What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets☆224Updated 10 months ago
- Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…☆131Updated 2 years ago
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆187Updated 2 months ago
- ☆194Updated 3 months ago
- ☆91Updated 3 years ago
- 💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023☆162Updated 4 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- A large-scale information-rich web dataset, featuring millions of real clicked query-document labels☆342Updated 9 months ago
- ☆15Updated 9 months ago
- Model implementation for the contextual embeddings project☆36Updated 4 months ago
- Pipeline for pulling and processing online language model pretraining data from the web☆177Updated 2 years ago
- Manage scalable open LLM inference endpoints in Slurm clusters☆274Updated last year
- ☆77Updated 9 months ago
- ☆111Updated 9 months ago
- Multipack distributed sampler for fast padding-free training of LLMs☆201Updated last year
- XTR: Rethinking the Role of Token Retrieval in Multi-Vector Retrieval☆58Updated last year
- ☆163Updated last year
- Baguetter is a flexible, efficient, and hackable search engine library implemented in Python. It's designed for quickly benchmarking, imp…☆189Updated last year
- ☆53Updated 2 months ago
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆49Updated last year
- Python API for https://vespa.ai, the open big data serving engine☆143Updated this week
- ☆62Updated last year
- multimodal document analysis☆167Updated last year
- Index of URLs to pdf files all over the internet and scripts☆24Updated 2 years ago
- FastFit ⚡ When LLMs are Unfit Use FastFit ⚡ Fast and Effective Text Classification with Many Classes☆211Updated 2 weeks ago