Alternatives and similar repositories for CommonCrawl:

Users that are interested in CommonCrawl are comparing it to the libraries listed below

• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆56Updated last year
• istresearch / traptor
  Traptor -- A distributed Twitter feed
  ☆26Updated 2 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated 9 months ago
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 3 years ago
• TeamHG-Memex / autologin-middleware
  Scrapy middleware for the autologin
  ☆37Updated 6 years ago
• charman / trawler
  Twitter crawler
  ☆11Updated 10 years ago
• matthewruttley / mozclassify
  Algorithms for URL Classification
  ☆19Updated 9 years ago
• yowenter / stackshare
  A simple Web crawler for stackshare.io using scrapy .
  ☆9Updated 6 years ago
• iproduct-database / vpm-filter-spark
  Virtual patent marking crawler at iproduct.epfl.ch
  ☆14Updated 7 years ago
• TeamHG-Memex / extract-html-diff
  extract difference between two html pages
  ☆32Updated 6 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 6 years ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• sammyer / BoilerPy
  Python port of Boilerpipe library
  ☆15Updated 6 years ago
• crawlbase / proxycrawl-node
  ProxyCrawl Node library for scraping and crawling
  ☆23Updated last year
• modocache / github-recommendation-engine
  Discover repositories you should be following on Github.
  ☆31Updated 12 years ago
• iandennismiller / gthnk
  Personal Knowledge Management System. Capture your ideas using plain old text files. Make a journal that lasts 100 years.
  ☆28Updated last year
• xtannier / WebAnnotator
  WebAnnotator is a tool for annotating Web pages. WebAnnotator is implemented as a Firefox extension (https://addons.mozilla.org/en-US/fi…
  ☆48Updated 3 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• xkxx / algodb
  Mad (╯°□°)╯'ing
  ☆10Updated 2 years ago
• scrapinghub / webpager
  Paginating the web
  ☆37Updated 11 years ago
• TeamHG-Memex / url-summary
  Show summary of a large number of URLs in a Jupyter Notebook
  ☆17Updated 3 years ago
• mbr / Flask-Debug
  A small debugging helper for Flask.
  ☆10Updated 8 years ago
• mediacloud / feed_seeker
  Find rss, atom, xml, and rdf feeds on webpages
  ☆30Updated 5 months ago
• AnthonySigogne / keyword-mining
  API - extract a list of keywords from a text.
  ☆18Updated 7 years ago
• machinebox / textbox_elastic_indexer
  Example how to pre-process news articles with textbox and index on Elastic Search
  ☆13Updated 7 years ago
• rkrzr / dataset-popular
  A dataset of popular pages (taken from <dir.yahoo.com>) with manually marked up semantic blocks.
  ☆15Updated 11 years ago
• nyimbi / dot_vim
  My dot files in one place - extensively edited over time. Your mileage may vary
  ☆2Updated 8 years ago
• trec-kba / streamcorpus-pipeline
  framework for making streamcorpus data
  ☆11Updated 8 years ago
• joelparkerhenderson / big-five-personality-traits
  Big Five personality traits: domains, aspects, facets
  ☆25Updated last year
• ecrmnn / builtwith
  Scrape data from BuiltWith.com
  ☆17Updated 7 years ago