Process Common Crawl data with Python and Spark
☆452Jan 20, 2026Updated last month
Alternatives and similar repositories for cc-pyspark
Users that are interested in cc-pyspark are comparing it to the libraries listed below
Sorting:
- Various Jupyter notebooks about Common Crawl data☆64Nov 22, 2025Updated 3 months ago
- News crawling with StormCrawler - stores content as WARC☆364Feb 19, 2025Updated last year
- Tools to construct and process Common Crawl webgraphs☆105Feb 19, 2026Updated 2 weeks ago
- Streaming WARC/ARC library for fast web archive IO☆451Dec 10, 2024Updated last year
- Tools to download and cleanup Common Crawl data☆1,039Apr 25, 2023Updated 2 years ago
- Useful tools to extract malayalam text from the Common Crawl Datasets☆28Dec 11, 2024Updated last year
- Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…☆135Jun 7, 2023Updated 2 years ago
- A robust web archive analytics toolkit☆132Oct 15, 2025Updated 4 months ago
- Common web archive utility code.☆62Updated this week
- Scientific articles using or citing Common Crawl data☆28Jan 9, 2026Updated last month
- An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.☆86Apr 21, 2021Updated 4 years ago
- CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop☆38Dec 17, 2024Updated last year
- Core Python Web Archiving Toolkit for replay and recording of web archives☆1,629Jan 21, 2026Updated last month
- super-Django-CC is a simle web interface for commoncrawl.org☆15Dec 8, 2022Updated 3 years ago
- ☆64Apr 9, 2024Updated last year
- Python 3 library for reading and writing warc files☆21Jan 29, 2018Updated 8 years ago
- ☆25Mar 20, 2024Updated last year
- Common Crawl fork of Apache Nutch☆40Updated this week
- Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database☆14Oct 26, 2021Updated 4 years ago
- Code used for sourcing and cleaning the BigScience ROOTS corpus☆319Mar 20, 2023Updated 2 years ago
- gzipstream allows Python to process multi-part gzip files from a streaming source☆23Feb 24, 2017Updated 9 years ago
- ☆1,636Apr 27, 2023Updated 2 years ago
- news-please - an integrated web crawler and information extractor for news that just works☆2,391Sep 21, 2025Updated 5 months ago
- Building the laion5B paper☆36May 6, 2022Updated 3 years ago
- Code for paper "Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging"☆16May 31, 2019Updated 6 years ago
- Python library for reading and writing warc files☆248Mar 7, 2022Updated 4 years ago
- GPU controlled Hetzner Cloud workers swarm for Crawling@Home project☆58Oct 9, 2022Updated 3 years ago
- Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.☆2,915Updated this week
- The RedPajama-Data repository contains code for preparing large datasets for training large language models.☆4,923Dec 7, 2024Updated last year
- Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…☆132Nov 21, 2025Updated 3 months ago
- Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.☆1,577Feb 24, 2026Updated last week
- 100k+ topic labeled news articles published from thousands of news websites☆19Aug 18, 2020Updated 5 years ago
- A distributed system for mining common crawl using SQS, AWS-EC2 and S3☆22Jun 24, 2014Updated 11 years ago
- ☆225Oct 27, 2025Updated 4 months ago
- Sort-friendly URI Reordering Transform (SURT) python module☆45Sep 11, 2025Updated 5 months ago
- ☆34Dec 10, 2021Updated 4 years ago
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,354Oct 27, 2025Updated 4 months ago
- Easily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...☆320Dec 9, 2023Updated 2 years ago
- Heuristic based boilerplate removal tool☆811Feb 25, 2025Updated last year