Alternatives and similar repositories for dirbot

Users that are interested in dirbot are comparing it to the libraries listed below

• julien-duponchelle / scrapy-elasticsearch
  A scrapy pipeline which send items to Elastic Search server
  ☆98Updated 7 years ago
• TeamHG-Memex / scrapy-dockerhub
  [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
  ☆11Updated 10 years ago
• tonywangcn / scaleable-crawler-with-docker-cluster
  a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
  ☆97Updated last year
• scrapinghub / docker-images
  ☆32Updated last year
• scrapinghub / scmongo
  MongoDB extensions for Scrapy
  ☆44Updated 10 years ago
• realpython / stack-spider
  ☆167Updated 6 years ago
• svola / fantasticsearch
  Fantasticsearch will provide various search-engine templates for ElasticSearch
  ☆36Updated 9 years ago
• rmax / scrapy-boilerplate
  Small set of utilities to simplify writing Scrapy spiders.
  ☆49Updated 9 years ago
• toumorokoshi / nlgen
  Natural Language Generator for Python
  ☆27Updated 8 years ago
• iammai / docker-scrapy-crawler
  docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".
  ☆42Updated 10 years ago
• TeamHG-Memex / autologin-middleware
  Scrapy middleware for the autologin
  ☆37Updated 7 years ago
• jxltom / scrapymon
  Simple Web UI for Scrapy spider management via Scrapyd
  ☆51Updated 6 years ago
• mmas / docker-scrapy-tor
  Scrapy environment with Tor for anonymous ip routing and Privoxy for http proxy
  ☆20Updated 8 years ago
• SEL-Columbia / bamboo
  Dynamic data analysis over the web. The logic to your data dashboards.
  ☆156Updated 10 years ago
• salimhamed / dashboard
  A simple dashboarding web application built using Flask and Bootstrap.
  ☆65Updated 2 years ago
• scrapinghub / crawlera-tools
  Crawlera tools
  ☆26Updated 9 years ago
• cantabular / scraperwiki-python
  ScraperWiki Python library for scraping and saving data
  ☆159Updated 2 years ago
• TeamHG-Memex / url-summary
  Show summary of a large number of URLs in a Jupyter Notebook
  ☆17Updated 3 years ago
• dataisbeautiful / scrapy-development-docker
  Web Scraping Development Docker Container
  ☆20Updated 9 years ago
• scrapinghub / testspiders
  Useful test spiders for Scrapy
  ☆185Updated 5 years ago
• rmax / dirbot-mysql
  Scrapy project based on dirbot to show how to use Twisted's adbapi to store the scraped data in MySQL.
  ☆117Updated 11 years ago
• 0neSe7en / jd-comments
  抓取京东商城的评论信息与商品信息. 过滤无价值的评论,并展示.
  ☆17Updated 8 years ago
• verginer / bnb_scrapy_tutorial
  A tutorial on how to write a scrapy spider to get data from Airbnb
  ☆29Updated 7 years ago
• albertsun / huginn-newsroom-scenarios
  Documentation and common scenarios for newsroom usage of https://github.com/cantino/huginn/
  ☆16Updated 8 years ago
• GovTechSG / micropurchase
  18F's micro-purchase threshold experiment management app.
  ☆7Updated 7 years ago
• johnjreiser / NJParcelTools
  Tools to work with New Jersey's cadastral spatial data. (Python and PostgreSQL)
  ☆17Updated last year
• haukurk / flask-restapi-recipe
  A recipe for a RESTful API with key authentication. Powered with Flask+SQLALchemy and a minimal amount of external dependencies.
  ☆59Updated 4 years ago
• charman / trawler
  Twitter crawler
  ☆11Updated 10 years ago
• corywalker / selenium-crawler
  Sometimes sites make crawling hard. Selenium-crawler uses selenium automation to fix that.
  ☆125Updated 12 years ago
• openstates / billy
  legacy backend for Open States
  ☆87Updated 5 years ago