dongrunhua/ScrapyUniversal external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

dongrunhua/ScrapyUniversal

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/dongrunhua/ScrapyUniversal)

Close

dongrunhua / ScrapyUniversalView on GitHubMore

• External links
• Readme badge

基于Scrapy的通用爬虫框架

☆25Jan 9, 2019Updated 7 years ago

Alternatives and similar repositories for ScrapyUniversal

Users that are interested in ScrapyUniversal are comparing it to the libraries listed below

• renyan97 / smart-home

  View on GitHub
  采用微信小程序来控制智能家居，包括数据采集显示，远程控制，蓝牙控制，语音控制等。
  ☆11Feb 19, 2019Updated 7 years ago
• sdogsq / ctrip-hotel-spider

  View on GitHub
  selenium 携程酒店爬虫+简单数据分析
  ☆10Dec 6, 2018Updated 7 years ago
• siseng / siseng.github.io

  View on GitHub
  homepage
  ☆10Feb 15, 2023Updated 3 years ago
• LoseUFaith / WOS_crawler

  View on GitHub
  利用 selenium 自动化控制 Chrome 浏览器以 Excel 格式导出 Web of Science 搜索结果
  ☆11Aug 15, 2022Updated 3 years ago
• lonycc / tools

  View on GitHub
  python 小工具
  ☆82Dec 2, 2021Updated 4 years ago
• zhixwang / Douban_backup

  View on GitHub
  豆瓣资料备份
  ☆13Sep 5, 2020Updated 5 years ago
• Baiyuetribe / watermark-img

  View on GitHub
  可能是全网最方便的水印图床，支持宝塔一键部署、也支持Docker版部署至服务器或本地电脑
  ☆10Jul 16, 2019Updated 6 years ago
• gxxgle / go-utils

  View on GitHub
  General utility functions for golang
  ☆13Sep 28, 2020Updated 5 years ago
• lnxpgn / scrapy_multiple_spiders

  View on GitHub
  Using multiple spiders in a Scrapy project
  ☆10Aug 7, 2015Updated 10 years ago
• henrylee123 / baiduIndexCrawler

  View on GitHub
  百度指数（百度热搜爬虫）（js破解版）
  ☆14Apr 9, 2019Updated 6 years ago
• imwalson / large-data-app-analysis

  View on GitHub
  大数据下的移动 APP 数据分析指南
  ☆13Jun 27, 2019Updated 6 years ago
• AaronJny / scrapy_redis_expiredupefilter

  View on GitHub
  scrapy-redis-expiredupefilter是基于scrapy-redis修改来的一款scrapy分布式爬虫框架，它支持为请求指纹设置生命周期，请求指纹生命周期结束后将在不影响其他指纹的情况下自动清除。
  ☆10Aug 6, 2019Updated 6 years ago
• doublekous / django_pays

  View on GitHub
  django框架中使用支付宝支付，微信native模式支付，中国银联支付
  ☆14Feb 7, 2023Updated 3 years ago
• ido-web / jianshu_spider

  View on GitHub
  Scrapy + selenium/webdriver + 随机User-Agent + IP proxy + twisted ConnectionPool + mysql 爬取某书整站爬虫
  ☆15Dec 8, 2022Updated 3 years ago
• 7rikka / python-bilibili-api

  View on GitHub
  bilibili-api
  ☆15Oct 18, 2018Updated 7 years ago
• lixi5338619 / lxparse

  View on GitHub
  用于解析列表页链接和提取详细页内容的库
  ☆20Oct 26, 2023Updated 2 years ago
• xdworks / speed-reading

  View on GitHub
  ☆14Dec 3, 2018Updated 7 years ago
• 1160007652 / douyin_download_m3u8

  View on GitHub
  下载抖音直播视频
  ☆14Mar 5, 2022Updated 4 years ago
• kingname / Tinepeas

  View on GitHub
  Tinepeas，我们自己的爬虫框架。
  ☆60Aug 9, 2024Updated last year
• guapier / zufang

  View on GitHub
  租房爬虫，基于flask，采用apscheduler定时任务，通过微信，定时给用户推送想要的租房信息
  ☆15Mar 13, 2019Updated 6 years ago
• yubang / FileSynchronization

  View on GitHub
  一个基于python的文件同步小工具
  ☆18May 29, 2015Updated 10 years ago
• 3519793075 / Project3

  View on GitHub
  电商产品评论数据情感分析模型
  ☆22Feb 21, 2023Updated 3 years ago
• Urinx / weChat_php

  View on GitHub
  自己微信公众号 Urinx 的后台
  ☆19Sep 9, 2015Updated 10 years ago
• liuyizhi123 / give-me-some-credit-master

  View on GitHub
  kaggle贷款违约预测 ：主要研究如何借助非平衡数据分类的思想对银行等金融机构的历史贷款数据进行分析，并基于随机森林分类模型预测贷款违约的可能性
  ☆23Apr 4, 2019Updated 6 years ago
• striver-ing / distributed-spider

  View on GitHub
  通用新闻类网站分布式爬虫
  ☆79Jul 17, 2018Updated 7 years ago
• HandsomeHan515 / python-wechat-pay

  View on GitHub
  Use Python3, Django, Django-rest-framework to achieve wechat payment. 微信支付、服务器异步通知、订单查询、退款
  ☆21Sep 21, 2018Updated 7 years ago
• huailiang / watermark

  View on GitHub
  去除抖音、微视水印的脚本工具
  ☆20Apr 30, 2020Updated 5 years ago
• duckzhao / yuanrenxue_js

  View on GitHub
  猿人学js攻防比赛的相关题目解答
  ☆24Sep 3, 2021Updated 4 years ago
• heyue-99 / Waterfall-Carousel-Accordion-Modal

  View on GitHub
  瀑布流布局 + 图片懒加载 / 轮播图 / 手风琴/ 模态框图片展示
  ☆14Aug 15, 2017Updated 8 years ago
• ctrip-moles / moles-packer

  View on GitHub
  moles-packer 是由携程框架团队研发的，与携程moles框架配套使用的React Native 打包和拆包工具，同时支持原生的 React Native 项目。
  ☆18Nov 24, 2016Updated 9 years ago
• wdc63 / chat-robot

  View on GitHub
  基于图灵机器人，科大讯飞win版，百度语音合成的语音聊天和音乐播放机器人。
  ☆22Jul 25, 2017Updated 8 years ago
• leffss / ScrapyRedisBloomFilterBlockCluster

  View on GitHub
  Scrapy Redis with Bloom Filter，support redis sentinel and cluster
  ☆25Mar 31, 2023Updated 2 years ago
• polestar-HF / Spider-project

  View on GitHub
  爬虫实战项目
  ☆25Feb 4, 2023Updated 3 years ago
• kingking888 / CommNewsExtractor

  View on GitHub
  通用文章提取，正文，标题，时间，作者，图片，音视频，联系方式等
  ☆23Mar 19, 2023Updated 2 years ago
• onesafe / m3u8_to_mp4

  View on GitHub
  convert m3u8 to mp4
  ☆20Aug 5, 2018Updated 7 years ago
• NiShuang / wx-index

  View on GitHub
  微信指数爬虫
  ☆23Aug 29, 2018Updated 7 years ago
• NearHuiwen / TaobaoCrawl

  View on GitHub
  淘宝根据搜索词搜索商品爬虫，无selenium，动态cookie，动态代理IP
  ☆24Dec 23, 2020Updated 5 years ago
• Postiii / twds-crawler

  View on GitHub
  Highly scalable webcrawler for towardsdatascience.com by using Python, Selenium, Docker, Kubernetes and the infrastructure of the Google …
  ☆25Jan 17, 2022Updated 4 years ago
• xinxinliang / ksDjango

  View on GitHub
  关于快手视频的django项目，之前使用爬虫，可以获取快手视频网站的视频，输入用户主页地址，就可以获取用户的视频地址、粉丝数、点赞数等。现在这个项目准备实现：随机获取用户id并且去重，然后根据id获取主页视频信息，将获取的内容展示到网页上。后期还可以开发用户注册登录系统，可…
  ☆21Mar 24, 2021Updated 4 years ago