Alternatives and similar repositories for awesome_crawl

Users that are interested in awesome_crawl are comparing it to the libraries listed below

• Yanxueshan / Scrapy-Redis-Zhihu

  View on GitHub
  基于scrapy-redis实现分布式爬虫，爬取知乎所有问题及对应的回答，集成selenium模拟登录、英文验证码及倒立文字验证码识别、随机生成User-Agent、IP代理、处理302重定向问题等等
  ☆61Apr 3, 2019Updated 6 years ago
• zhangslob / docs

  View on GitHub
  《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用n…
  ☆138Jun 26, 2019Updated 6 years ago
• yinzishao / NewsScrapy

  View on GitHub
  基于scrapy的新闻爬虫
  ☆101Apr 18, 2020Updated 5 years ago
• Mocha-Pudding / Scrapy-Redis_Demos

  View on GitHub
  Python分布式爬虫学习笔记，各种Demo同步
  ☆12Aug 21, 2019Updated 6 years ago
• Jaysong2012 / tutorial

  View on GitHub
  Scrapy爬虫实战系列，从零开始爬取腾讯百度淘宝知乎各大网站内容 \n 12306刷票脚本系列
  ☆82Apr 2, 2019Updated 6 years ago
• chenjiandongx / bili-spider

  View on GitHub
  📺 B 站全站视频信息爬虫
  ☆680Feb 17, 2019Updated 6 years ago
• HEUDavid / WeiboSpider

  View on GitHub
  微博爬虫 有问题欢迎提出来
  ☆17Jul 2, 2019Updated 6 years ago
• caiyongji / wechat-ranking

  View on GitHub
  微信公众号10w+文章数据
  ☆35Sep 7, 2018Updated 7 years ago
• TauWu / weibo_daily_hotkey

  View on GitHub
  Weibo's daily TOP5 hotkey. 自动爬取、筛选新浪微博每日热搜词 TOP5。https://github.com/TauWu/weibo_daily_hotkey/blob/master/data/data.md
  ☆36Apr 18, 2021Updated 4 years ago
• ex34toway / uw-log-es

  View on GitHub
  一个基于ElasticSearch的业务日志记录工具
  ☆10Nov 5, 2018Updated 7 years ago
• jfzhang95 / news_spider

  View on GitHub
  新闻爬虫 (腾讯,网易,新浪,今日头条,搜狐,凤凰网,腾讯滚动新闻)
  ☆58Jun 6, 2018Updated 7 years ago
• Python3WebSpider / DouYin

  View on GitHub
  API of DouYin for Humans used to Crawl Popular Videos and Musics
  ☆651Jan 29, 2020Updated 6 years ago
• shisiying / tc_zufang

  View on GitHub
  使用scrapy,redis, mongodb,django实现的一个分布式网络爬虫,底层存储mongodb,分布式使用redis实现,使用django可视化爬虫
  ☆282May 1, 2018Updated 7 years ago
• airingursb / bilibili-video

  View on GitHub
  🍙 Bilibili 视频爬虫
  ☆324Jun 18, 2022Updated 3 years ago
• linian365boy / rule-core

  View on GitHub
  规则引擎
  ☆22Feb 28, 2018Updated 7 years ago
• wnma3mz / wechat_articles_spider

  View on GitHub
  微信公众号文章的爬虫
  ☆3,355Apr 18, 2024Updated last year
• whbrewer / spc

  View on GitHub
  The Scientific Platform for the Cloud (SPC) - migrate and run scientific applications in the cloud
  ☆10Feb 7, 2026Updated last week
• facert / tumblr_spider

  View on GitHub
  汤不热 python 多线程爬虫
  ☆462Jul 22, 2020Updated 5 years ago
• shikanon / proxy_scrapy

  View on GitHub
  proxy_scrapy是一个scrapy搭建的代理模块，主要包括代理抓取、代理测试和使用代理三个模块。包括了对主要的代理网站的抓取和代理稳定性的测试，并整合进scrapy爬虫当中。
  ☆10Jan 20, 2017Updated 9 years ago
• Gerapy / GerapyProxy

  View on GitHub
  A package for supporting proxy in Scrapy & Gerapy
  ☆11Jul 15, 2020Updated 5 years ago
• CrackerCat / CallObfuscator

  View on GitHub
  Obfuscate specific windows apis with different apis
  ☆12Jul 31, 2024Updated last year
• madneal / OSCP_learing

  View on GitHub
  ☆14May 22, 2024Updated last year
• HegemonyTao / crawlProject

  View on GitHub
  今日头条 、淘宝 、微博 、斗鱼 、抖音 、哔哩哔哩 、有道翻译、steam网站以及网易云音乐爬取
  ☆61Apr 17, 2020Updated 5 years ago
• weweibuy / gateway

  View on GitHub
  Springcloud gateway 的样例工程, 封装了一些常用的网关功能,如鉴权,数据级权限控制,验签,动态路由,日志,流控,降级等
  ☆21Jul 8, 2022Updated 3 years ago
• AlexTan-b-z / ZhihuSpider

  View on GitHub
  知乎分布式爬虫（Scrapy、Redis）
  ☆168Feb 18, 2018Updated 7 years ago
• nghuyong / WeiboSpider

  View on GitHub
  持续维护的新浪微博采集工具🚀🚀🚀
  ☆4,029Aug 23, 2025Updated 5 months ago
• Wooden-Robot / scrapy-tutorial

  View on GitHub
  Scrapy 爬虫框架教程源码
  ☆108Aug 23, 2019Updated 6 years ago
• bowenpay / wechat-spider

  View on GitHub
  微信公众号爬虫
  ☆3,295Aug 10, 2021Updated 4 years ago
• davideuler / goproxy

  View on GitHub
  基于go写的隧道代理服务器，主要用于翻墙。
  ☆12Nov 20, 2016Updated 9 years ago
• virusdefender / ssrf-app

  View on GitHub
  ☆14Jan 3, 2020Updated 6 years ago
• orangeMask / spider

  View on GitHub
  抖音,淘宝系,常见新闻爬虫
  ☆13Apr 15, 2022Updated 3 years ago
• hadesvip / ly-security

  View on GitHub
  业务风控系统
  ☆12Aug 25, 2016Updated 9 years ago
• starryrbs / awesome-scrapy

  View on GitHub
  scrapy实战教程，分享scrapy爬虫的知识，针对各大网站做爬虫采集，并且以实例代码讲解。
  ☆11Jan 22, 2026Updated 3 weeks ago
• WYL-BruceLong / ProxyPool

  View on GitHub
  爬虫代理池的维护
  ☆10Oct 9, 2018Updated 7 years ago
• sundyloveme / GameCore

  View on GitHub
  能快速开发出游戏的游戏引擎，本来是如鹏网的GameCore游戏引擎，我对它封装，便于自己用
  ☆11Feb 14, 2016Updated 10 years ago
• kong36088 / ZhihuSpider

  View on GitHub
  多线程知乎用户爬虫，基于python3
  ☆249May 29, 2023Updated 2 years ago
• starFalll / Spider

  View on GitHub
  新浪微博爬虫(Sina weibo spider)，百度搜索结果 爬虫
  ☆197Jul 17, 2023Updated 2 years ago
• lb2281075105 / Python-Spider

  View on GitHub
  豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github…
  ☆783Aug 27, 2022Updated 3 years ago
• chyroc / WechatSogou

  View on GitHub
  基于搜狗微信搜索的微信公众号爬虫接口
  ☆6,181Nov 15, 2023Updated 2 years ago