Alternatives and similar repositories for web-crawler:

Users that are interested in web-crawler are comparing it to the libraries listed below

• kenshinx / second-spider
  one more spider based on gevent requests pyquery
  ☆54Updated 10 years ago
• Greyh4t / ProxyPool
  Crawl and validate proxies from Internet
  ☆77Updated 8 years ago
• eastrd / HighAnonProxyPool
  A proxy pool that scrapes free anonymous proxies and maintains its proxies' availability.
  ☆93Updated 7 years ago
• akun / PyHackerNews
  Hacker News written in Python
  ☆17Updated 9 years ago
• pokemonish / chrome_remote_interface_python
  Chrome Debugging Protocol interface for Python
  ☆110Updated 6 years ago
• VillanCh / scraplat
  I'm trying to finish the scraplat as a scraper platform
  ☆48Updated 9 years ago
• GeeTeam / gt-python-sdk
  GtWeb Python Sdk
  ☆82Updated 7 years ago
• devhub / domaintools
  Domain parsing with Python
  ☆44Updated 3 weeks ago
• Masutangu / Elric
  Elric: A Simple Distributed Job Scheduler
  ☆85Updated 8 years ago
• tanjiti / perl_tools
  perl小工具
  ☆30Updated 7 years ago
• kbdancer / wooyun_rank
  采集乌云已确认漏洞和已公开漏洞的状态、厂商、Rank等数据用于分析哪些是良心厂商
  ☆14Updated 8 years ago
• LeoHuang2015 / ops_security
  some info about ops security(system, network, etc)
  ☆28Updated 9 years ago
• reorx / deptest
  dependent testing framework
  ☆15Updated 7 years ago
• Hongxs / scrapy-sebug
  利用scrapy框架抓取sebug漏洞详情页
  ☆13Updated 10 years ago
• Tr3jer / AutoHookSpider
  将自动爬虫的结果判断是否属于hooks，并不断抓取url爬啊爬。
  ☆31Updated 7 years ago
• Pynow / elasticsearch
  elasticsearch Python脚本
  ☆28Updated 7 years ago
• wilson9x1 / crawler_phantomjs
  ☆14Updated 8 years ago
• OneSourceCat / scan-framework
  A framework used for Vulnerability scanning
  ☆106Updated 9 years ago
• VillanCh / pr0xy
  教学用多线程扫描框架（代理扫描）
  ☆39Updated 8 years ago
• qiaofei32 / yunCrawler
  智能云爬虫Demo
  ☆32Updated 7 years ago
• weizetao / spider-roach
  分布式定向抓取集群
  ☆71Updated 7 years ago
• HatBoy / RTask
  Python3+Huey+Zerorpc+Redis+Flask=RTask 轻量级分布式任务管理系统
  ☆74Updated 8 years ago
• scrapinghub / wappalyzer-python
  UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)
  ☆82Updated 8 years ago
• hillsbird / githubscan
  根据关键字扫描github代码泄露
  ☆11Updated 6 years ago
• ysrc / ChromeExtensionSpider
  Google Extension WebStore 爬虫，crx文件下载和内容解析 By Nearg1e
  ☆26Updated 8 years ago
• yinghuocho / ghttproxy
  a gevent based HTTP/HTTPS proxy with SOCKS5 forwarding.
  ☆32Updated 8 years ago
• nanshihui / normal_hark_lite
  通用的POC检测框架，有足够的POC，就可以找出相应的漏洞
  ☆44Updated 8 years ago
• julien-duponchelle / scrapy-graphite
  Output scrapy statistics to graphite/carbon
  ☆54Updated 12 years ago
• yubang / cms
  一个基于flask的cms系统
  ☆12Updated 10 years ago
• x0day / MultiProxies
  penetration testing framework that can use socks4/socks5 proxy.
  ☆53Updated 10 years ago