Alternatives and similar repositories for crawlerflow

Users that are interested in crawlerflow are comparing it to the libraries listed below

• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated this week
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆58Updated 2 years ago
• codelucas / shorten.tv
  Python video summarization. Visit the public API at -- www.shorten.tv (EDIT: The domain expired and youtube blocked it ..)
  ☆84Updated 3 years ago
• devvid / python-common-crawl-amazon-example
  Exploring Common-Crawl using Python and DynamoDB
  ☆33Updated 8 years ago
• scrapinghub / scaws
  Extensions for using Scrapy on Amazon AWS
  ☆32Updated 13 years ago
• pschwede / AnchorBot
  The more often you click a word in the headlines, the more interesting are your news.
  ☆13Updated 8 years ago
• tonywangcn / scaleable-crawler-with-docker-cluster
  a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
  ☆97Updated last year
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated 2 weeks ago
• SocialHarvest / harvester
  The Social Harvest server that exposes an API and harvests data from the web to be analyzed.
  ☆115Updated 10 years ago
• crawlerflow / crawlerflow
  Orchestrate web crawlers to create structured datasets from multiple data sources with YAML configs.
  ☆15Updated 3 years ago
• istresearch / traptor
  Traptor -- A distributed Twitter feed
  ☆26Updated 3 years ago
• psolbach / mehrwert
  Aviation grade news article metadata extraction
  ☆36Updated 2 years ago
• Parsely / serpextract
  Easy extraction of keywords and engines from search engine results pages (SERPs).
  ☆93Updated 3 months ago
• DansProjects / airflow-averageface
  Creates a pipeline Airflow and Scrapy to output an average image composition of everyone's face in a given website
  ☆44Updated 8 years ago
• maxdemarzi / neo_graph_search
  A POC at replicating Facebook Graph Search with Cypher and Neo4j
  ☆101Updated 12 years ago
• isaacmg / fb_scraper
  FBLYZE is a Facebook scraping system and analysis system.
  ☆67Updated 4 years ago
• NUKnightLab / twxplorer
  ☆36Updated 2 years ago
• DEKHTIARJonathan / FeedCrunch.IO
  FeedCrunch.IO - Take RSS Feeds to the next level with personnalized recommendations
  ☆15Updated 3 years ago
• openeventdata / scraper
  Scrapes sites. Gets news. Eventually events.
  ☆85Updated 9 years ago
• bahn / wikitopics
  Exploits Wikipedia's daily view counts to find out what topics are current trends
  ☆18Updated 12 years ago
• iammai / docker-scrapy-crawler
  docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".
  ☆42Updated 10 years ago
• PatrickJS / Reddit-Insight
  Reddit Analytics built in Angular.js
  ☆141Updated 11 years ago
• melvynator / ELK_twitter
  This is a data pipeline for Twitter (ETL) using the elastic stack Elasticsearch, Logstash and Kibana (version 6.1)
  ☆59Updated 7 years ago
• gfjreg / CommonCrawl
  A distributed system for mining common crawl using SQS, AWS-EC2 and S3
  ☆21Updated 11 years ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆193Updated 3 years ago
• Riamse / ceterach
  An interface for interacting with MediaWiki
  ☆38Updated 4 years ago
• abhishekbalam / keeplinkin
  A Flask and Redis based fast, feature rich and free URL shortener site.
  ☆23Updated 3 years ago
• svola / ElasticUI-extension
  An extension to the demo template of ElasticUI a beautiful AngularJS frontend to ElasticSearch for faceted navigation
  ☆39Updated 10 years ago
• pferrel / solr-recommender
  ☆16Updated 9 years ago
• opensemanticsearch / open-semantic-etl
  Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & N…
  ☆277Updated 3 years ago