Alternatives and similar repositories for web-crawler:

Users that are interested in web-crawler are comparing it to the libraries listed below

• kenshinx / second-spider
  one more spider based on gevent requests pyquery
  ☆54Updated 10 years ago
• Greyh4t / ProxyPool
  Crawl and validate proxies from Internet
  ☆77Updated 8 years ago
• eastrd / HighAnonProxyPool
  A proxy pool that scrapes free anonymous proxies and maintains its proxies' availability.
  ☆93Updated 7 years ago
• HatBoy / RTask
  Python3+Huey+Zerorpc+Redis+Flask=RTask 轻量级分布式任务管理系统
  ☆74Updated 8 years ago
• Masutangu / Elric
  Elric: A Simple Distributed Job Scheduler
  ☆85Updated 8 years ago
• devhub / domaintools
  Domain parsing with Python
  ☆44Updated last month
• wrfly / captcha_verify
  简单验证码识别
  ☆28Updated 9 years ago
• VillanCh / scraplat
  I'm trying to finish the scraplat as a scraper platform
  ☆48Updated 9 years ago
• julien-duponchelle / scrapy-graphite
  Output scrapy statistics to graphite/carbon
  ☆54Updated 12 years ago
• zrools / spider
  ☆114Updated 8 years ago
• leitro / knowsecSpider2
  知道创宇爬虫题目 持续更新版本
  ☆94Updated 10 years ago
• qiaofei32 / yunCrawler
  智能云爬虫Demo
  ☆32Updated 7 years ago
• TeamHG-Memex / arachnado
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆162Updated 2 years ago
• flisky / scrapy-phantomjs-downloader
  PhantomJS Downloader for Scrapy, Yeah!
  ☆94Updated 10 years ago
• weizetao / spider-roach
  分布式定向抓取集群
  ☆71Updated 7 years ago
• GeeTeam / gt-python-sdk
  GtWeb Python Sdk
  ☆82Updated 7 years ago
• LiuRoy / spider_docker
  为爬虫引用创建container，包括的模块：scrapy, mongo, celery, rabbitmq
  ☆36Updated 9 years ago
• 2shou / PhantomjsFetcher
  A python web fetcher using phantomjs to mock browser
  ☆180Updated 7 years ago
• pokemonish / chrome_remote_interface_python
  Chrome Debugging Protocol interface for Python
  ☆110Updated 6 years ago
• akun / PyHackerNews
  Hacker News written in Python
  ☆17Updated 9 years ago
• yinghuocho / ghttproxy
  a gevent based HTTP/HTTPS proxy with SOCKS5 forwarding.
  ☆32Updated 9 years ago
• BruceDone / tensorflow-demos
  all kinds of demos of tensorflow code
  ☆96Updated 7 years ago
• howie6879 / hproxy
  hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)
  ☆66Updated 3 years ago
• wuchong / scrapy-dynamic-configurable
  A dynamic configurable news crawler based Scrapy
  ☆166Updated 7 years ago
• xelzmm / proxy_server_crawler
  an awesome public proxy server crawler based on scrapy framework
  ☆96Updated 7 years ago
• lopuhin / scrapy-pyppeteer
  Use pyppeteer from a Scrapy spider
  ☆59Updated 5 years ago
• scrapinghub / wappalyzer-python
  UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)
  ☆82Updated 8 years ago
• ioiogoo / scrapy-monitor
  scrapy-monitor，实现爬虫可视化，监控实时状态
  ☆109Updated 8 years ago
• BruceDone / scrapy_demo
  all kinds of scrapy demo
  ☆164Updated 2 years ago
• kbdancer / wooyun_rank
  采集乌云已确认漏洞和已公开漏洞的状态、厂商、Rank等数据用于分析哪些是良心厂商
  ☆14Updated 8 years ago