Alternatives and similar repositories for crawlerflow

Users that are interested in crawlerflow are comparing it to the libraries listed below

• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 4 years ago
• psolbach / metadoc
  Aviation grade news article metadata extraction
  ☆36Updated 2 years ago
• devvid / python-common-crawl-amazon-example
  Exploring Common-Crawl using Python and DynamoDB
  ☆33Updated 7 years ago
• tonywangcn / scaleable-crawler-with-docker-cluster
  a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
  ☆97Updated last year
• iammai / docker-scrapy-crawler
  docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".
  ☆42Updated 10 years ago
• jondot / pgpipeline
  A Scrapy pipeline module to persist items to a postgres table automatically.
  ☆21Updated 8 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated last year
• DEKHTIARJonathan / FeedCrunch.IO
  FeedCrunch.IO - Take RSS Feeds to the next level with personnalized recommendations
  ☆15Updated 3 years ago
• bahn / wikitopics
  Exploits Wikipedia's daily view counts to find out what topics are current trends
  ☆18Updated 12 years ago
• melvynator / ELK_twitter
  This is a data pipeline for Twitter (ETL) using the elastic stack Elasticsearch, Logstash and Kibana (version 6.1)
  ☆59Updated 7 years ago
• tpeng / googlesearch
  Scrape the Google search result with Scrapy.
  ☆99Updated 5 years ago
• codelucas / shorten.tv
  Python video summarization. Visit the public API at -- www.shorten.tv (EDIT: The domain expired and youtube blocked it ..)
  ☆81Updated 3 years ago
• crawlerflow / crawlerflow
  Orchestrate web crawlers to create structured datasets from multiple data sources with YAML configs.
  ☆14Updated 2 years ago
• DansProjects / airflow-averageface
  Creates a pipeline Airflow and Scrapy to output an average image composition of everyone's face in a given website
  ☆44Updated 7 years ago
• maxdemarzi / neo_graph_search
  A POC at replicating Facebook Graph Search with Cypher and Neo4j
  ☆101Updated 12 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆55Updated last year
• TeamHG-Memex / autologin-middleware
  Scrapy middleware for the autologin
  ☆36Updated 7 years ago
• abhishekbalam / keeplinkin
  A Flask and Redis based fast, feature rich and free URL shortener site.
  ☆23Updated 2 years ago
• TeamHG-Memex / deep-deep
  Adaptive crawler which uses Reinforcement Learning methods
  ☆169Updated 7 years ago
• superdesk / superdesk-core
  Common server modules for superdesk
  ☆31Updated last week
• SocialHarvest / harvester
  The Social Harvest server that exposes an API and harvests data from the web to be analyzed.
  ☆115Updated 10 years ago
• NUKnightLab / twxplorer
  ☆36Updated last year
• guillermo-carrasco / social_ids
  Get user ids from social network handlers
  ☆12Updated 8 years ago
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 8 years ago
• dbs / postgresql-full-text-search-engine
  This application demonstrates how to use PostgreSQL as a full-text search engine.
  ☆63Updated 7 years ago
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆119Updated last year
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated 2 years ago
• pferrel / solr-recommender
  ☆16Updated 8 years ago
• Sotera / newman
  Quickly analyze and explore email with advanced analytics and visualization.
  ☆56Updated 3 years ago