Alternatives and similar repositories for web-crawler

Users that are interested in web-crawler are comparing it to the libraries listed below

• eastrd / HighAnonProxyPool
  A proxy pool that scrapes free anonymous proxies and maintains its proxies' availability.
  ☆94Updated 7 years ago
• Greyh4t / ProxyPool
  Crawl and validate proxies from Internet
  ☆78Updated 8 years ago
• TeamHG-Memex / arachnado
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆161Updated 2 years ago
• pokemonish / chrome_remote_interface_python
  Chrome Debugging Protocol interface for Python
  ☆109Updated 7 years ago
• kenshinx / second-spider
  one more spider based on gevent requests pyquery
  ☆54Updated 11 years ago
• xelzmm / proxy_server_crawler
  an awesome public proxy server crawler based on scrapy framework
  ☆93Updated 8 years ago
• Pynow / elasticsearch
  elasticsearch Python脚本
  ☆28Updated 7 years ago
• zrools / spider
  ☆113Updated 9 years ago
• qiaofei32 / yunCrawler
  智能云爬虫Demo
  ☆32Updated 8 years ago
• scrapinghub / wappalyzer-python
  UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)
  ☆82Updated 8 years ago
• VillanCh / scraplat
  I'm trying to finish the scraplat as a scraper platform
  ☆48Updated 9 years ago
• devhub / domaintools
  Domain parsing with Python
  ☆44Updated 6 months ago
• 2shou / PhantomjsFetcher
  A python web fetcher using phantomjs to mock browser
  ☆180Updated 7 years ago
• h01 / ProxyScanner
  python HTTP代理扫描
  ☆127Updated 10 years ago
• ysrc / ChromeExtensionSpider
  Google Extension WebStore 爬虫，crx文件下载和内容解析 By Nearg1e
  ☆27Updated 8 years ago
• Masutangu / Elric
  Elric: A Simple Distributed Job Scheduler
  ☆86Updated 9 years ago
• BruceDone / scrapy_demo
  all kinds of scrapy demo
  ☆164Updated 2 years ago
• binux / pyproxy
  HTTP(s) proxy server base on tornado, in one file!
  ☆107Updated 4 years ago
• relip / python-whois
  A whois library that retrieves and parses whois data.
  ☆25Updated 9 years ago
• BruceDone / Tiktok
  Python web visualize build on the awesome web framework sanic
  ☆62Updated 8 years ago
• weizetao / spider-roach
  分布式定向抓取集群
  ☆71Updated 8 years ago
• restran / web-proxy
  ⛺️ A reverse proxy for web site based on Tornado
  ☆53Updated 8 years ago
• netxfly / dianying
  中国主流在线电影网站爬虫及搜索web代码
  ☆34Updated 11 years ago
• flisky / scrapy-phantomjs-downloader
  PhantomJS Downloader for Scrapy, Yeah!
  ☆94Updated 11 years ago
• kbdancer / wooyun_rank
  采集乌云已确认漏洞和已公开漏洞的状态、厂商、Rank等数据用于分析哪些是良心厂商
  ☆14Updated 8 years ago
• lcatro / Aurora_CAPTCHA
  极光验证码,为反爬虫而生
  ☆27Updated 5 years ago
• mysterymask / wooyun
  wooyun.org公开漏洞爬取
  ☆23Updated 8 years ago
• meibenjin / weibo_crawler
  微博搜索结果爬取工具
  ☆27Updated 10 years ago
• wrfly / captcha_verify
  简单验证码识别
  ☆28Updated 9 years ago
• jiehua233 / ipproxy
  代理IP提取工具
  ☆116Updated 8 years ago