Alternatives and similar repositories for crawlerflow

Users that are interested in crawlerflow are comparing it to the libraries listed below

• crawlerflow / crawlerflow
  Orchestrate web crawlers to create structured datasets from multiple data sources with YAML configs.
  ☆14Updated 2 years ago
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 4 years ago
• pferrel / solr-recommender
  ☆16Updated 8 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• iammai / docker-scrapy-crawler
  docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".
  ☆42Updated 10 years ago
• istresearch / traptor
  Traptor -- A distributed Twitter feed
  ☆26Updated 2 years ago
• psolbach / metadoc
  Aviation grade news article metadata extraction
  ☆36Updated 2 years ago
• DansProjects / airflow-averageface
  Creates a pipeline Airflow and Scrapy to output an average image composition of everyone's face in a given website
  ☆44Updated 7 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated last year
• julien-duponchelle / scrapy-elasticsearch
  A scrapy pipeline which send items to Elastic Search server
  ☆98Updated 7 years ago
• scrapinghub / scaws
  Extensions for using Scrapy on Amazon AWS
  ☆32Updated 12 years ago
• TeamHG-Memex / scrapy-dockerhub
  [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
  ☆11Updated 10 years ago
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 8 years ago
• matthewruttley / mozclassify
  Algorithms for URL Classification
  ☆19Updated 10 years ago
• gfjreg / CommonCrawl
  A distributed system for mining common crawl using SQS, AWS-EC2 and S3
  ☆21Updated 11 years ago
• Sotera / newman
  Quickly analyze and explore email with advanced analytics and visualization.
  ☆56Updated 3 years ago
• VIDA-NYU / domain_discovery_tool
  This repository contains the Domain Discovery Tool (DDT) project. DDT is an interactive system that helps users explore and better unders…
  ☆45Updated 3 years ago
• mitll / topic-clustering
  ☆43Updated 9 years ago
• matejbasic / PythonScrapyBasicSetup
  Basic setup with random user agents and IP addresses for Python Scrapy Framework.
  ☆58Updated 7 years ago
• ClimbsRocks / empythy
  Automated NLP sentiment predictions- batteries included, or use your own data
  ☆18Updated 7 years ago
• verifid / ner-d
  Python module for Named Entity Recognition (NER) using natural language processing.
  ☆13Updated 4 years ago
• bahn / wikitopics
  Exploits Wikipedia's daily view counts to find out what topics are current trends
  ☆17Updated 12 years ago
• jondot / pgpipeline
  A Scrapy pipeline module to persist items to a postgres table automatically.
  ☆21Updated 7 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 7 years ago
• commoncrawl / gzipstream
  gzipstream allows Python to process multi-part gzip files from a streaming source
  ☆23Updated 8 years ago
• svola / fantasticsearch
  Fantasticsearch will provide various search-engine templates for ElasticSearch
  ☆36Updated 9 years ago
• ogrisel / dbpediakit
  Python utilities to do work with the DBpedia dumps for analytics.
  ☆39Updated 13 years ago
• plouc / flambo
  The open-source content aggregation platform.
  ☆13Updated 8 years ago
• devvid / python-common-crawl-amazon-example
  Exploring Common-Crawl using Python and DynamoDB
  ☆33Updated 7 years ago
• ChrisRoark / beagle_scraper
  Scraper for categories and lists on ecommerce and other listing websites
  ☆42Updated 4 years ago