Alternatives and similar repositories for crawlerflow

Users that are interested in crawlerflow are comparing it to the libraries listed below

• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated this week
• devvid / python-common-crawl-amazon-example
  Exploring Common-Crawl using Python and DynamoDB
  ☆33Updated 8 years ago
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 4 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• DEKHTIARJonathan / FeedCrunch.IO
  FeedCrunch.IO - Take RSS Feeds to the next level with personnalized recommendations
  ☆15Updated 3 years ago
• tonywangcn / scaleable-crawler-with-docker-cluster
  a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
  ☆97Updated last year
• crawlerflow / crawlerflow
  Orchestrate web crawlers to create structured datasets from multiple data sources with YAML configs.
  ☆14Updated 3 years ago
• codelucas / shorten.tv
  Python video summarization. Visit the public API at -- www.shorten.tv (EDIT: The domain expired and youtube blocked it ..)
  ☆81Updated 3 years ago
• maxdemarzi / neo_graph_search
  A POC at replicating Facebook Graph Search with Cypher and Neo4j
  ☆101Updated 12 years ago
• jondot / pgpipeline
  A Scrapy pipeline module to persist items to a postgres table automatically.
  ☆21Updated 8 years ago
• plumi / plumi.app
  plumi video sharing
  ☆113Updated 6 years ago
• matejbasic / PythonScrapyBasicSetup
  Basic setup with random user agents and IP addresses for Python Scrapy Framework.
  ☆58Updated 8 years ago
• Parsely / serpextract
  Easy extraction of keywords and engines from search engine results pages (SERPs).
  ☆92Updated 2 months ago
• abhishekbalam / keeplinkin
  A Flask and Redis based fast, feature rich and free URL shortener site.
  ☆23Updated 3 years ago
• zone-eu / zone-mta-template
  Template application for ZoneMTA
  ☆42Updated this week
• cldellow / real-estate-prices-cc
  Source real estate prices from the Common Crawl.
  ☆27Updated 7 years ago
• superdesk / newsroom
  Superdesk Newshub a.k.a. Newsroom
  ☆23Updated last year
• pschwede / AnchorBot
  The more often you click a word in the headlines, the more interesting are your news.
  ☆13Updated 8 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆55Updated 2 years ago
• haridas / IP-monitoring
  Monitor your IP reputation for Email sending or Email marketing.
  ☆43Updated 12 years ago
• theju / smp
  Social Media Post scheduler
  ☆23Updated 8 years ago
• guillermo-carrasco / social_ids
  Get user ids from social network handlers
  ☆12Updated 8 years ago
• openbridge / ob_bulkstash
  Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either …
  ☆115Updated 5 years ago
• bahn / wikitopics
  Exploits Wikipedia's daily view counts to find out what topics are current trends
  ☆18Updated 12 years ago
• mgmacias95 / TwitterFriends
  Django app that makes a graph dataset with your friends on twitter
  ☆36Updated 8 years ago
• openeventdata / scraper
  Scrapes sites. Gets news. Eventually events.
  ☆85Updated 9 years ago
• dcondrey / scrapy-spiders
  Collection of python scripts I have created to crawl various websites, mostly for lead generation projects to match keywords and collect …
  ☆134Updated 2 years ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• jesseward / musicdiscovery-assist
  Google Home assistant for music recommendations, built with Python & Flask. Using Google Home and API.ai
  ☆15Updated 8 years ago
• scrapinghub / scaws
  Extensions for using Scrapy on Amazon AWS
  ☆32Updated 13 years ago