Distributed crawling/scraping, Kafka And Redis based components for Scrapy
☆45Nov 13, 2020Updated 5 years ago
Alternatives and similar repositories for scrapy-kafka-redis
Users that are interested in scrapy-kafka-redis are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Kafka-based components for Scrapy☆78Apr 10, 2018Updated 8 years ago
- scrapy-redis-expiredupefilter是基于scrapy-redis修改来的一款scrapy分布式爬虫框架，它支持为请求指纹设置生命周期，请求指纹生命周期结束后将在不影响其他指纹的情况下自动清除。☆10Aug 6, 2019Updated 7 years ago
- Spider项目将会不断更新本人学习使用过的爬虫方法！！！☆15Aug 27, 2017Updated 9 years ago
- Scrapy and Kafka☆14Feb 7, 2018Updated 8 years ago
- 个人博客☆13Feb 2, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Nintendo Switch 云游戏！☆12May 8, 2020Updated 6 years ago
- Elastic Beat that consumes Apache Kafka topics and ships events to Elasticsearch, Logstash or any Beats output. Plain and JSON codecs.☆12Apr 29, 2019Updated 7 years ago
- 一个快速确定文本（新闻）归属地的工具☆18Sep 29, 2020Updated 6 years ago
- 公众号文章代码☆62Feb 14, 2019Updated 7 years ago
- ☆11Jun 14, 2020Updated 6 years ago
- A RabbitMQ Scheduler for Scrapy☆86Aug 9, 2022Updated 4 years ago
- 难用的frida通用脚本工具☆48Jul 31, 2025Updated last year
- A modern implementation of IRC server in Go☆18Sep 12, 2014Updated 12 years ago
- 以太坊钱包助记词顺序恢复工具☆17Aug 4, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- nsq官方文档翻译和源码学习☆11Sep 10, 2014Updated 12 years ago
- Scrapy Selenium on Taobao Product☆86Aug 6, 2017Updated 9 years ago
- Stream Nginx logs directly into InfluxDB☆14Sep 22, 2017Updated 9 years ago
- 在scrapyd基础上新增权限验证、爬虫运行信息统计、界面重构、，并增加排序、筛选过滤等多个API☆111Nov 6, 2018Updated 8 years ago
- Let's Build A Template Engine（让我们一起来构建一个模板引擎）☆12Jul 10, 2016Updated 10 years ago
- 基于Redis的Bloomfilter去重，并将其扩展到Scrapy框架。☆345Feb 26, 2023Updated 3 years ago
- ☆17Jul 14, 2017Updated 9 years ago
- Use scrapy with mongodb to store the request queues (FIFO or LIFO)☆13Dec 25, 2012Updated 13 years ago
- Use WeChat to connect Claude Code☆26Apr 12, 2026Updated 6 months ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- 在内网中使用代理上互联网☆10Mar 7, 2023Updated 3 years ago
- HTML DOM Query Language for XGo☆42Feb 16, 2026Updated 8 months ago
- 一款优秀的在线文件预览解决方案，，使用主流springboot+maven搭建，支持doc、docx、ppt、pptx、xls、xlsx、zip、rar、mp4、mp3以及众多类文本如txt、html、xml、java、properties、sql、js、md、json、c…☆16Dec 6, 2022Updated 3 years ago
- ☆12Apr 4, 2024Updated 2 years ago
- A chrome extension to get XPath of list items in webpage easily.☆34Mar 11, 2022Updated 4 years ago
- Helm Chart for Samba4☆10Oct 13, 2021Updated 5 years ago
- Auto Extractor Module☆335Aug 19, 2024Updated 2 years ago
- Scrapy Universal Spider☆57Aug 26, 2017Updated 9 years ago
- 这是一个基于js的加密解密算法项目☆13Nov 18, 2022Updated 3 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Scrapy Tutorial☆50Dec 12, 2021Updated 4 years ago
- ScrapyDemo : Redis MySQLdb logging IngoreHttpRequestMiddleware UserAgentMiddleware HttpProxyMiddleware rules☆37Jun 28, 2016Updated 10 years ago
- 基于darknet实现目标检测，提供识别点选验证码的实例和训练自己数据的API☆403Nov 22, 2022Updated 3 years ago
- 书籍《Python3 反爬虫原理与绕过实战》配套代码☆625Oct 25, 2021Updated 5 years ago
- Scrapy Pyppeteer Demo☆24Jul 13, 2018Updated 8 years ago
- Extension for BlackSheep that simplifies the use of SQLAlchemy in the web framework.☆17Mar 27, 2025Updated last year
- 可视化任务调度系统，精简到一个二进制文件 (Web visual task scheduler system , yes ! just one binary solve all the problems !)☆196Mar 21, 2026Updated 7 months ago