Alternatives and similar repositories for CommonCrawl:

Users that are interested in CommonCrawl are comparing it to the libraries listed below

• istresearch / traptor
  Traptor -- A distributed Twitter feed
  ☆26Updated 2 years ago
• TeamHG-Memex / autologin-middleware
  Scrapy middleware for the autologin
  ☆37Updated 6 years ago
• ClimbsRocks / empythy
  Automated NLP sentiment predictions- batteries included, or use your own data
  ☆18Updated 7 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆56Updated last year
• datamade / scrapers-us-municipal
  Scrapers for US municipal governments.
  ☆10Updated last year
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• ContinuumIO / scrapy_scrapers
  Scraper built with Scrapy.
  ☆15Updated 7 months ago
• yowenter / stackshare
  A simple Web crawler for stackshare.io using scrapy .
  ☆9Updated 6 years ago
• xkxx / algodb
  Mad (╯°□°)╯'ing
  ☆10Updated 2 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 6 years ago
• USCDataScience / autoextractor
  A toolkit for clustering web pages based on various similarity measures.
  ☆33Updated 3 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆54Updated last year
• iproduct-database / vpm-filter-spark
  Virtual patent marking crawler at iproduct.epfl.ch
  ☆14Updated 7 years ago
• xtannier / WebAnnotator
  WebAnnotator is a tool for annotating Web pages. WebAnnotator is implemented as a Firefox extension (https://addons.mozilla.org/en-US/fi…
  ☆48Updated 3 years ago
• verifid / ner-d
  Python module for Named Entity Recognition (NER) using natural language processing.
  ☆13Updated 3 years ago
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 3 years ago
• scrapinghub / webpager
  Paginating the web
  ☆37Updated 11 years ago
• scrapinghub / scaws
  Extensions for using Scrapy on Amazon AWS
  ☆32Updated 12 years ago
• matthewruttley / mozclassify
  Algorithms for URL Classification
  ☆19Updated 9 years ago
• TeamHG-Memex / url-summary
  Show summary of a large number of URLs in a Jupyter Notebook
  ☆17Updated 3 years ago
• TeamHG-Memex / fortia
  [UNMAINTAINED] Firefox addon for Scrapely
  ☆5Updated 9 years ago
• guillermo-carrasco / social_ids
  Get user ids from social network handlers
  ☆12Updated 8 years ago
• machinebox / textbox_elastic_indexer
  Example how to pre-process news articles with textbox and index on Elastic Search
  ☆13Updated 7 years ago
• MKLab-ITI / reveal-user-classification
  Performs user classification into labels using a set of seed Twitter users with known labels and the structure of the interaction network…
  ☆10Updated 8 years ago
• Parsely / python-nlp-slides
  Slides to learn a little natural language processing (NLP) with Python. Written in reST with S5/Docutils.
  ☆28Updated 12 years ago
• aritter / LDA-SP
  Includes Code for Inference and Evaluation of Topic Models for Selectional Preferences
  ☆16Updated 2 years ago
• shavit / crawlero
  Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
  ☆16Updated 7 years ago
• rgalanakis / pynocle
  Code metrics for Python code.
  ☆11Updated 10 years ago
• src-d / apollo
  Advanced similarity and duplicate source code proof of concept for our research efforts.
  ☆52Updated 2 years ago
• stefanbehr / sumtract
  Second project for UW LING 572. Automatic text summarization system.
  ☆13Updated 12 years ago