Alternatives and similar repositories for crawler

Users that are interested in crawler are comparing it to the libraries listed below

• atupal / ccrawler
  A distrubuted crawler ues celery.
  ☆17Updated 10 years ago
• papaya-mobile / cyber
  A distributed asynchronous socket framework of Python
  ☆13Updated 8 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated 11 months ago
• shish / devtools-py
  A Python client for Chrome's DevTools protocol / a headless chrome control library
  ☆15Updated 6 years ago
• kakwa / pygraph_redis
  Simple python library to manipulate directed graphs in redis
  ☆23Updated 7 years ago
• TeamHG-Memex / scrapy-dockerhub
  [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
  ☆11Updated 10 years ago
• gtoonstra / remap
  MapReduce platform in python
  ☆34Updated 9 years ago
• arkaitzj / wsvnc
  In browser VNC client through websockets
  ☆30Updated 13 years ago
• scrapy-plugins / scrapy-pagestorage
  A scrapy extension to store requests and responses information in storage service
  ☆26Updated 3 years ago
• redapple / parslepy
  Python implementation of the Parsley language for extracting structured data from web pages
  ☆92Updated 7 years ago
• datalib / StatsCounter
  Python's missing statistical Swiss Army knife
  ☆15Updated 9 years ago
• marklit / mass-ipv4-whois
  Mass IPv4 WHOIS Collection Tool
  ☆18Updated 9 years ago
• zodb / perfmetrics
  A library for sending software performance metrics from Python libraries and apps to statsd.
  ☆30Updated 11 months ago
• acdha / python-phash
  ctypes bindings for libphash to robustly compare media files
  ☆12Updated 2 years ago
• relip / python-whois
  A whois library that retrieves and parses whois data.
  ☆25Updated 9 years ago
• scrapinghub / aduana
  Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even whe…
  ☆55Updated 11 months ago
• ssteuteville / scrapyz
  "Scrape Easy" - an extension of the Scrapy framework.
  ☆187Updated 8 years ago
• bbrodriges / pholcidae
  Tiny python web crawler
  ☆169Updated 9 years ago
• stefanw / scrapa
  Python 3 AsyncIO powered scraping framework with batteries included
  ☆20Updated 8 years ago
• getdnsapi / getdns-python-bindings
  Python bindings for getdns
  ☆25Updated 4 years ago
• collinalexbell / Email-Autoresponder
  ☆21Updated 10 years ago
• twaldear / flask-secure-headers
  Secure Header Wrapper for Flask Applications
  ☆18Updated 5 years ago
• jvansteirteghem / twunnel3
  A HTTPS/SOCKS4/SOCKS5 tunnel for AsyncIO.
  ☆21Updated 9 years ago
• discoproject / discodb
  An efficient, immutable, persistent mapping object
  ☆98Updated 6 years ago
• crowdwave / serve_swagger
  Use Swagger API JSON to create Falcon REST APIs.
  ☆32Updated 8 years ago
• ceteri / slinky
  Slinky, a high-performance web crawler / text analytics in Python, Redis, Hadoop, R, Gephi
  ☆41Updated 14 years ago
• atmb4u / butler
  Python Dictionaries and Lists on Steroids
  ☆23Updated 10 years ago
• TeamHG-Memex / autologin
  A project to attempt to automatically login to a website given a single seed
  ☆124Updated 2 years ago
• andrewjw / celery-crawler
  A Django based search engine powered by CouchDB, celery and whoosh.
  ☆49Updated 9 years ago
• ivannotes / yoton
  Add cache support for python function with less pain
  ☆9Updated 2 years ago