commoncrawl / cc-downloaderLinks
A polite and user-friendly downloader for Common Crawl data
☆51Updated last week
Alternatives and similar repositories for cc-downloader
Users that are interested in cc-downloader are comparing it to the libraries listed below
Sorting:
- A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine☆183Updated 7 months ago
- Statistics of Common Crawl monthly archives mined from URL index files☆189Updated last week
- Next-generation Punkt sentence boundary detection with zero dependencies☆17Updated 3 weeks ago
- Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …☆51Updated last week
- Tools to construct and process Common Crawl webgraphs☆93Updated last week
- A sentence segmentation library with wide language support optimized for speed and utility.☆66Updated 2 months ago
- Common crawl extractor☆78Updated last year
- An index of PDF-centric corpora☆137Updated last month
- Fast and robust date extraction from web pages, with Python or on the command-line☆138Updated 3 weeks ago
- Streaming WARC/ARC library for fast web archive IO☆428Updated 8 months ago
- Small python package to measure OCR quality and other related metrics.☆25Updated last year
- 🔢 Work with static vector models☆28Updated 4 months ago
- Libraries, Archives and Museums (LAM)☆85Updated 2 years ago
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆46Updated 7 years ago
- ☆67Updated last year
- Code for collecting, processing, and preparing datasets for the Common Pile☆221Updated this week
- Faster, modernized fork of the language identification tool langid.py☆56Updated 9 months ago
- Efficiently computing & storing token n-grams from large corpora☆26Updated 10 months ago
- Targetted language identifier, based on FastText and Hunspell.☆37Updated 6 months ago
- Benchmark scripts for comparing different tokenizers and sentence segmenters of German☆12Updated 2 years ago
- A set of utilities for processing MediaWiki XML dump data.☆57Updated 6 months ago
- Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters☆143Updated 7 months ago
- A helper library full of URL-related heuristics.☆70Updated 2 months ago
- State-of-the-art web crawler 🔱☆305Updated this week
- A tool for detecting viruses and NSFW material in WARC files☆15Updated last year
- WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.☆258Updated 6 months ago
- search interface for scholarly works☆86Updated last year
- A Memento Aggregator CLI and Server in Go☆68Updated 5 months ago
- Command line tool for digging into WARC files☆45Updated last week
- LLM plugin for embeddings using sentence-transformers☆70Updated 4 months ago