Alternatives and similar repositories for Distributed-Web-Crawler-with-Celery

Users that are interested in Distributed-Web-Crawler-with-Celery are comparing it to the libraries listed below

• kadnan / ScrapeGen
  A simple python tool that generates a requests/bs4 based web scraper
  ☆27Updated 3 years ago
• TeamHG-Memex / autologin-middleware
  Scrapy middleware for the autologin
  ☆37Updated 7 years ago
• scrapinghub / arche
  Analyze scraped data
  ☆46Updated 5 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated last year
• ncouture / python-search-engine
  Search engine base (crawler, indexer and parser) using Python, Celery, RabbitMQ, CouchDB and Whoosh.
  ☆11Updated 2 weeks ago
• gwpicard / flask-kanban
  A Flask-powered Kanban app
  ☆10Updated 4 years ago
• scrapy-plugins / scrapy-querycleaner
  Scrapy spider middleware to clean up query parameters in request URLs
  ☆24Updated 8 years ago
• TeamHG-Memex / scrapy-kafka-export
  Scrapy extension which writes crawled items to Kafka
  ☆30Updated 6 years ago
• gbnk0 / simple-image-classifier
  Simple image classifier microservice using tensorflow and sanic
  ☆25Updated 6 years ago
• scrapy-plugins / scrapy-jsonschema
  Scrapy schema validation pipeline and Item builder using JSON Schema
  ☆44Updated 4 years ago
• adipasquale / techcrunch-incremental-scrapy-spider-with-mongodb
  Techcrunch Incremental Scrapy Spider With MongoDB
  ☆16Updated 6 years ago
• lethain / extraction
  A Python library for extracting titles, images, descriptions and canonical urls from HTML.
  ☆150Updated 5 years ago
• jxltom / scrapymon
  Simple Web UI for Scrapy spider management via Scrapyd
  ☆51Updated 7 years ago
• scrapy-plugins / scrapy-headless
  ☆29Updated 4 years ago
• scrapy-plugins / scrapy-pagestorage
  A scrapy extension to store requests and responses information in storage service
  ☆26Updated 3 years ago
• lopuhin / scrapy-pyppeteer
  Use pyppeteer from a Scrapy spider
  ☆59Updated 5 years ago
• scrapinghub / scrapy-autoextract
  Zyte Automatic Extraction integration for Scrapy
  ☆56Updated 3 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 7 years ago
• scrapinghub / scrapy-mosquitera
  Restrict crawl and scraping scope using matchers.
  ☆26Updated 9 years ago
• colmex / frontera_example
  Example frontera project
  ☆12Updated 7 years ago
• DormyMo / scrappy
  scrapy best practice
  ☆38Updated 4 years ago
• anuragrana / scraping_tweets_celery_rabbitmq_docker_cluster
  Scraping tweets quickly using celery, RabbitMQ and Docker cluster
  ☆48Updated 2 years ago
• TeamHG-Memex / scrapy-dockerhub
  [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
  ☆11Updated 10 years ago
• ponyriders / django-amazon-price-monitor
  Monitors prices of Amazon products via Product Advertising API
  ☆156Updated 6 years ago
• anuragrana / celery_rabbitmq_docker_cluster
  boilerplate code to start with celery and rabbitmq in docker cluster
  ☆20Updated 2 years ago
• AndreiD / SlackUptimeMonitor
  Receive notifications in Slack when your websites/api/services are down
  ☆16Updated 4 years ago
• priestc / python-etsy
  Python library for interacting with the Etsy API
  ☆15Updated 3 years ago
• udger / udger-python
  Python agent string parser based on Udger https://udger.com/products/local_parser
  ☆40Updated last year
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• stefanw / scrapa
  Python 3 AsyncIO powered scraping framework with batteries included
  ☆20Updated 8 years ago