Alternatives and similar repositories for cc-pyspark

Users that are interested in cc-pyspark are comparing it to the libraries listed below

• commoncrawl / cc-index-table

  View on GitHub
  Index Common Crawl archives in tabular format
  ☆125Dec 22, 2025Updated last month
• commoncrawl / cc-notebooks

  View on GitHub
  Various Jupyter notebooks about Common Crawl data
  ☆62Nov 22, 2025Updated 2 months ago
• commoncrawl / cc-mrjob

  View on GitHub
  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework
  ☆168Jan 27, 2026Updated 2 weeks ago
• commoncrawl / news-crawl

  View on GitHub
  News crawling with StormCrawler - stores content as WARC
  ☆363Feb 19, 2025Updated 11 months ago
• commoncrawl / cc-webgraph

  View on GitHub
  Tools to construct and process Common Crawl webgraphs
  ☆105Jan 30, 2026Updated 2 weeks ago
• webrecorder / warcio

  View on GitHub
  Streaming WARC/ARC library for fast web archive IO
  ☆445Dec 10, 2024Updated last year
• commoncrawl / cdx_toolkit

  View on GitHub
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆198Jan 23, 2026Updated 3 weeks ago
• facebookresearch / cc_net

  View on GitHub
  Tools to download and cleanup Common Crawl data
  ☆1,039Apr 25, 2023Updated 2 years ago
• ikreymer / cdx-index-client

  View on GitHub
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆205Oct 7, 2018Updated 7 years ago
• shjwudp / c4-dataset-script

  View on GitHub
  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…
  ☆135Jun 7, 2023Updated 2 years ago
• chatnoir-eu / chatnoir-resiliparse

  View on GitHub
  A robust web archive analytics toolkit
  ☆130Oct 15, 2025Updated 3 months ago
• commoncrawl / commoncrawl-examples

  View on GitHub
  A library of examples showing how to use the Common Crawl corpus (2008-2012, ARC format)
  ☆65Aug 5, 2016Updated 9 years ago
• trivio / common_crawl_index

  View on GitHub
  Index URLs in Common Crawl
  ☆198Sep 19, 2017Updated 8 years ago
• commoncrawl / cc-citations

  View on GitHub
  Scientific articles using or citing Common Crawl data
  ☆28Jan 9, 2026Updated last month
• oscar-project / goclassy

  View on GitHub
  An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.
  ☆86Apr 21, 2021Updated 4 years ago
• commoncrawl / cc-warc-examples

  View on GitHub
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆37Dec 17, 2024Updated last year
• webrecorder / pywb

  View on GitHub
  Core Python Web Archiving Toolkit for replay and recording of web archives
  ☆1,619Jan 21, 2026Updated 3 weeks ago
• devvid / python-common-crawl-amazon-example

  View on GitHub
  Exploring Common-Crawl using Python and DynamoDB
  ☆33Oct 26, 2017Updated 8 years ago
• imfht / super-Django-CC

  View on GitHub
  super-Django-CC is a simle web interface for commoncrawl.org
  ☆15Dec 8, 2022Updated 3 years ago
• ilinguistics / common_crawl_corpus

  View on GitHub
  Scripts for building a geo-located web corpus using Common Crawl data
  ☆11Jan 18, 2026Updated 3 weeks ago
• locuslab / scaling_laws_data_filtering

  View on GitHub
  ☆64Apr 9, 2024Updated last year
• rossf7 / elasticrawl

  View on GitHub
  Launch AWS Elastic MapReduce jobs that process Common Crawl data.
  ☆49Feb 15, 2017Updated 8 years ago
• jpbruinsslot / warc3

  View on GitHub
  Python 3 library for reading and writing warc files
  ☆21Jan 29, 2018Updated 8 years ago
• commoncrawl / nutch

  View on GitHub
  Common Crawl fork of Apache Nutch
  ☆40Updated this week
• dkpro / dkpro-c4corpus

  View on GitHub
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Jun 12, 2020Updated 5 years ago
• oscar-project / ungoliant

  View on GitHub
  The pipeline for the OSCAR corpus
  ☆176Nov 9, 2025Updated 3 months ago
• ikreymer / webarchive-indexing

  View on GitHub
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆47Dec 4, 2017Updated 8 years ago
• bigscience-workshop / data-preparation

  View on GitHub
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆318Mar 20, 2023Updated 2 years ago
• commoncrawl / gzipstream

  View on GitHub
  gzipstream allows Python to process multi-part gzip files from a streaming source
  ☆23Feb 24, 2017Updated 8 years ago
• EleutherAI / the-pile

  View on GitHub
  ☆1,636Apr 27, 2023Updated 2 years ago
• fhamborg / news-please

  View on GitHub
  news-please - an integrated web crawler and information extractor for news that just works
  ☆2,382Sep 21, 2025Updated 4 months ago
• LAION-AI / laion5B-paper

  View on GitHub
  Building the laion5B paper
  ☆36May 6, 2022Updated 3 years ago
• cisnlp / semi-markov-crf

  View on GitHub
  Code for paper "Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging"
  ☆16May 31, 2019Updated 6 years ago
• internetarchive / warc

  View on GitHub
  Python library for reading and writing warc files
  ☆247Mar 7, 2022Updated 3 years ago
• huggingface / datatrove

  View on GitHub
  Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
  ☆2,885Updated this week
• togethercomputer / RedPajama-Data

  View on GitHub
  The RedPajama-Data repository contains code for preparing large datasets for training large language models.
  ☆4,924Dec 7, 2024Updated last year
• ChenghaoMou / text-dedup

  View on GitHub
  All-in-one text de-duplication
  ☆742Jan 2, 2026Updated last month
• ukwa / webarchive-discovery

  View on GitHub
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆132Nov 21, 2025Updated 2 months ago
• internetarchive / surt

  View on GitHub
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆44Sep 11, 2025Updated 5 months ago