zhangslob/awesome_crawl external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

zhangslob/awesome_crawl

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/zhangslob/awesome_crawl)

Close

zhangslob / awesome_crawlView on GitHubMore

• External links
• Readme badge

腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

☆302Jun 6, 2025Updated 9 months ago

Alternatives and similar repositories for awesome_crawl

Users that are interested in awesome_crawl are comparing it to the libraries listed below

• zyingzhou / zuiyouSpider

  View on GitHub
  最右APP爬虫，用Python爬取最右APP段子数据和视频弹幕。
  ☆21Jun 29, 2019Updated 6 years ago
• zhangslob / docs

  View on GitHub
  《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用n…
  ☆138Jun 26, 2019Updated 6 years ago
• yinzishao / NewsScrapy

  View on GitHub
  基于scrapy的新闻爬虫
  ☆101Apr 18, 2020Updated 5 years ago
• Jacen789 / NewsCrawler

  View on GitHub
  新闻爬虫，爬取新浪、搜狐、新华网即时财经新闻。
  ☆192May 9, 2020Updated 5 years ago
• Mocha-Pudding / Scrapy-Redis_Demos

  View on GitHub
  Python分布式爬虫学习笔记，各种Demo同步
  ☆12Aug 21, 2019Updated 6 years ago
• Jannchie / simpyder

  View on GitHub
  超高速异步协程Python爬虫
  ☆80Feb 15, 2023Updated 3 years ago
• Jaysong2012 / tutorial

  View on GitHub
  Scrapy爬虫实战系列，从零开始爬取腾讯百度淘宝知乎各大网站内容 \n 12306刷票脚本系列
  ☆81Apr 2, 2019Updated 6 years ago
• chenjiandongx / bili-spider

  View on GitHub
  📺 B 站全站视频信息爬虫
  ☆685Feb 17, 2019Updated 7 years ago
• HEUDavid / WeiboSpider

  View on GitHub
  微博爬虫 有问题欢迎提出来
  ☆17Jul 2, 2019Updated 6 years ago
• TauWu / weibo_daily_hotkey

  View on GitHub
  Weibo's daily TOP5 hotkey. 自动爬取、筛选新浪微博每日热搜词 TOP5。https://github.com/TauWu/weibo_daily_hotkey/blob/master/data/data.md
  ☆36Apr 18, 2021Updated 4 years ago
• caiyongji / wechat-ranking

  View on GitHub
  微信公众号10w+文章数据
  ☆36Sep 7, 2018Updated 7 years ago
• Randy-whiteSugar / LagouSpider_Scrapy

  View on GitHub
  使用Scrapy编写的拉勾网爬虫，添加了代理IP池、增量爬取机制
  ☆11May 22, 2023Updated 2 years ago
• ex34toway / uw-log-es

  View on GitHub
  一个基于ElasticSearch的业务日志记录工具
  ☆10Nov 5, 2018Updated 7 years ago
• jfzhang95 / news_spider

  View on GitHub
  新闻爬虫 (腾讯,网易,新浪,今日头条,搜狐,凤凰网,腾讯滚动新闻)
  ☆58Jun 6, 2018Updated 7 years ago
• Python3WebSpider / DouYin

  View on GitHub
  API of DouYin for Humans used to Crawl Popular Videos and Musics
  ☆651Jan 29, 2020Updated 6 years ago
• shisiying / tc_zufang

  View on GitHub
  使用scrapy,redis, mongodb,django实现的一个分布式网络爬虫,底层存储mongodb,分布式使用redis实现,使用django可视化爬虫
  ☆282May 1, 2018Updated 7 years ago
• 97CBR / SoftwareUpdateServer

  View on GitHub
  Software Update Server 软件更新服务
  ☆22Jul 9, 2024Updated last year
• airingursb / bilibili-video

  View on GitHub
  🍙 Bilibili 视频爬虫
  ☆325Jun 18, 2022Updated 3 years ago
• linian365boy / rule-core

  View on GitHub
  规则引擎
  ☆22Feb 28, 2018Updated 8 years ago
• wnma3mz / wechat_articles_spider

  View on GitHub
  微信公众号文章的爬虫
  ☆3,369Apr 18, 2024Updated last year
• facert / tumblr_spider

  View on GitHub
  汤不热 python 多线程爬虫
  ☆464Jul 22, 2020Updated 5 years ago
• Gerapy / GerapyProxy

  View on GitHub
  A package for supporting proxy in Scrapy & Gerapy
  ☆11Jul 15, 2020Updated 5 years ago
• CrackerCat / CallObfuscator

  View on GitHub
  Obfuscate specific windows apis with different apis
  ☆12Jul 31, 2024Updated last year
• madneal / OSCP_learing

  View on GitHub
  ☆14May 22, 2024Updated last year
• shikanon / proxy_scrapy

  View on GitHub
  proxy_scrapy是一个scrapy搭建的代理模块，主要包括代理抓取、代理测试和使用代理三个模块。包括了对主要的代理网站的抓取和代理稳定性的测试，并整合进scrapy爬虫当中。
  ☆10Jan 20, 2017Updated 9 years ago
• HegemonyTao / crawlProject

  View on GitHub
  今日头条 、淘宝 、微博 、斗鱼 、抖音 、哔哩哔哩 、有道翻译、steam网站以及网易云音乐爬取
  ☆61Apr 17, 2020Updated 5 years ago
• AlexTan-b-z / ZhihuSpider

  View on GitHub
  知乎分布式爬虫（Scrapy、Redis）
  ☆168Feb 18, 2018Updated 8 years ago
• nghuyong / WeiboSpider

  View on GitHub
  持续维护的新浪微博采集工具🚀🚀🚀
  ☆4,040Aug 23, 2025Updated 6 months ago
• Wooden-Robot / scrapy-tutorial

  View on GitHub
  Scrapy 爬虫框架教程源码
  ☆108Aug 23, 2019Updated 6 years ago
• bowenpay / wechat-spider

  View on GitHub
  微信公众号爬虫
  ☆3,305Aug 10, 2021Updated 4 years ago
• yangge11 / scrapy_pro

  View on GitHub
  关于5000+站点的scrapy爬虫开发，涉及一些技术架构搭建以及各种反爬方案，详见readme文件
  ☆30Dec 8, 2022Updated 3 years ago
• WYL-BruceLong / ProxyPool

  View on GitHub
  爬虫代理池的维护
  ☆10Oct 9, 2018Updated 7 years ago
• orangeMask / spider

  View on GitHub
  抖音,淘宝系,常见新闻爬虫
  ☆13Apr 15, 2022Updated 3 years ago
• starryrbs / awesome-scrapy

  View on GitHub
  scrapy实战教程，分享scrapy爬虫的知识，针对各大网站做爬虫采集，并且以实例代码讲解。
  ☆11Jan 22, 2026Updated last month
• virusdefender / ssrf-app

  View on GitHub
  ☆14Jan 3, 2020Updated 6 years ago
• sundyloveme / GameCore

  View on GitHub
  能快速开发出游戏的游戏引擎，本来是如鹏网的GameCore游戏引擎，我对它封装，便于自己用
  ☆11Feb 14, 2016Updated 10 years ago
• davideuler / goproxy

  View on GitHub
  基于go写的隧道代理服务器，主要用于翻墙。
  ☆12Nov 20, 2016Updated 9 years ago
• kong36088 / ZhihuSpider

  View on GitHub
  多线程知乎用户爬虫，基于python3
  ☆249May 29, 2023Updated 2 years ago
• starFalll / Spider

  View on GitHub
  新浪微博爬虫(Sina weibo spider)，百度搜索结果 爬虫
  ☆197Jul 17, 2023Updated 2 years ago