Alternatives and similar repositories for CommonCrawl

Users that are interested in CommonCrawl are comparing it to the libraries listed below

• istresearch / traptor
  Traptor -- A distributed Twitter feed
  ☆26Updated 2 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated last year
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• TeamHG-Memex / extract-html-diff
  extract difference between two html pages
  ☆32Updated 7 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆56Updated last year
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 4 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 7 years ago
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 2 years ago
• charman / trawler
  Twitter crawler
  ☆11Updated 10 years ago
• scrapinghub / page_finder
  Find which links on a web page are pagination links
  ☆29Updated 8 years ago
• rmdort / spellchecker
  Spell correct entire sentences using nltk freqdist and symspell
  ☆19Updated 7 years ago
• iproduct-database / vpm-filter-spark
  Virtual patent marking crawler at iproduct.epfl.ch
  ☆14Updated 7 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆54Updated last year
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• yowenter / stackshare
  A simple Web crawler for stackshare.io using scrapy .
  ☆9Updated 6 years ago
• ClimbsRocks / empythy
  Automated NLP sentiment predictions- batteries included, or use your own data
  ☆18Updated 7 years ago
• matthewruttley / mozclassify
  Algorithms for URL Classification
  ☆19Updated 10 years ago
• ContinuumIO / scrapy_scrapers
  Scraper built with Scrapy.
  ☆17Updated 9 months ago
• nik0spapp / sdalg
  Web page segmentation and noise removal
  ☆55Updated last year
• TeamHG-Memex / url-summary
  Show summary of a large number of URLs in a Jupyter Notebook
  ☆17Updated 3 years ago
• invana / crawlerflow
  Web Crawlers orchestration framework that lets you create datasets from multiple web sources using yaml configurations.
  ☆34Updated last year
• Kagami / github-social-graph
  Build simple social graphs for GitHub
  ☆15Updated 10 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• stefanbehr / sumtract
  Second project for UW LING 572. Automatic text summarization system.
  ☆13Updated 12 years ago
• scrapinghub / autopager
  Detect and classify pagination links
  ☆15Updated 4 years ago
• EpistasisLab / ml-analyst
  Analysis pipeline for quick ML analyses.
  ☆11Updated 6 years ago
• scrapinghub / docker-images
  ☆32Updated last year
• semanticize / st
  Semanticizest: dump parser and client
  ☆20Updated 9 years ago
• scrapinghub / webpager
  Paginating the web
  ☆37Updated 11 years ago
• AnthonySigogne / keyword-mining
  API - extract a list of keywords from a text.
  ☆18Updated 7 years ago