Alternatives and similar repositories for web-crawler

Users that are interested in web-crawler are comparing it to the libraries listed below

• eastrd / HighAnonProxyPool
  A proxy pool that scrapes free anonymous proxies and maintains its proxies' availability.
  ☆92Updated 8 years ago
• Greyh4t / ProxyPool
  Crawl and validate proxies from Internet
  ☆78Updated 9 years ago
• xelzmm / proxy_server_crawler
  an awesome public proxy server crawler based on scrapy framework
  ☆91Updated 8 years ago
• TeamHG-Memex / arachnado
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆160Updated last week
• VillanCh / scraplat
  I'm trying to finish the scraplat as a scraper platform
  ☆48Updated 9 years ago
• kenshinx / second-spider
  one more spider based on gevent requests pyquery
  ☆53Updated 11 years ago
• pokemonish / chrome_remote_interface_python
  Chrome Debugging Protocol interface for Python
  ☆108Updated 7 years ago
• scrapinghub / wappalyzer-python
  UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)
  ☆82Updated 8 years ago
• devhub / domaintools
  Domain parsing with Python
  ☆44Updated 10 months ago
• phith0n / python-xss-filter
  Based on native Python module HTMLParser purifier of HTML, To Clear all javascript in html
  ☆116Updated 9 years ago
• Masutangu / Elric
  Elric: A Simple Distributed Job Scheduler
  ☆85Updated 9 years ago
• Pynow / elasticsearch
  elasticsearch Python脚本
  ☆28Updated 8 years ago
• qiaofei32 / yunCrawler
  智能云爬虫Demo
  ☆32Updated 8 years ago
• zrools / spider
  ☆112Updated 9 years ago
• kbdancer / wooyun_rank
  采集乌云已确认漏洞和已公开漏洞的状态、厂商、Rank等数据用于分析哪些是良心厂商
  ☆14Updated 9 years ago
• weizetao / spider-roach
  分布式定向抓取集群
  ☆71Updated 8 years ago
• leitro / knowsecSpider2
  知道创宇爬虫题目 持续更新版本
  ☆94Updated 11 years ago
• LeoHuang2015 / ops_security
  some info about ops security(system, network, etc)
  ☆28Updated 10 years ago
• h01 / ProxyScanner
  python HTTP代理扫描
  ☆125Updated 11 years ago
• meibenjin / weibo_crawler
  微博搜索结果爬取工具
  ☆27Updated 11 years ago
• julien-duponchelle / scrapy-graphite
  Output scrapy statistics to graphite/carbon
  ☆54Updated 12 years ago
• ring04h / fpymysql
  A Friendly pymysql CURD Class
  ☆75Updated 6 years ago
• akun / PyHackerNews
  Hacker News written in Python
  ☆18Updated 9 years ago
• HatBoy / RTask
  Python3+Huey+Zerorpc+Redis+Flask=RTask 轻量级分布式任务管理系统
  ☆74Updated 9 years ago
• senko / tornado-proxy
  Asyncronous HTTP proxy with tunnelling (CONNECT) support
  ☆333Updated 2 years ago
• 2shou / PhantomjsFetcher
  A python web fetcher using phantomjs to mock browser
  ☆180Updated 8 years ago
• nanshihui / normal_hark_lite
  通用的POC检测框架，有足够的POC，就可以找出相应的漏洞
  ☆45Updated 9 years ago
• manwu91 / inst_proxy
  批量安装squid代理
  ☆13Updated 7 years ago
• h-j-13 / Malicious_Domain_Whois
  非法域名挖掘与画像系统
  ☆80Updated 7 years ago
• restran / web-proxy
  ⛺️ A reverse proxy for web site based on Tornado
  ☆53Updated 9 years ago