Distributed crawling/scraping, Kafka And Redis based components for Scrapy
☆46Nov 13, 2020Updated 5 years ago
Alternatives and similar repositories for scrapy-kafka-redis
Users that are interested in scrapy-kafka-redis are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Kafka-based components for Scrapy☆78Apr 10, 2018Updated 8 years ago
- Scrapy and Kafka☆14Feb 7, 2018Updated 8 years ago
- Nintendo Switch 云游戏！☆12May 8, 2020Updated 6 years ago
- nsq官方文档翻译和源码学习☆11Sep 10, 2014Updated 11 years ago
- A RabbitMQ Scheduler for Scrapy☆87Aug 9, 2022Updated 3 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- 一个基于 HttpCanary 和 Python 的爬虫项目☆21May 2, 2023Updated 3 years ago
- Stream Nginx logs directly into InfluxDB☆14Sep 22, 2017Updated 8 years ago
- A Simple Tool to Distribute/Administrate Your Scripts☆12Sep 10, 2015Updated 10 years ago
- requests升级版requests-html 爬虫编写及通用爬虫模块搭建☆11Nov 21, 2018Updated 7 years ago
- ☆17Jul 14, 2017Updated 8 years ago
- 基于Redis的Bloomfilter去重，并将其扩展到Scrapy框架。☆347Feb 26, 2023Updated 3 years ago
- Let's Build A Template Engine（让我们一起来构建一个模板引擎）☆12Jul 10, 2016Updated 9 years ago
- my greasemonkey scripts☆18May 13, 2022Updated 4 years ago
- Auto Extractor Module☆338Aug 19, 2024Updated last year
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Scrapy + Puppeteer☆110Jun 11, 2021Updated 5 years ago
- 可视化任务调度系统，精简到一个二进制文件 (Web visual task scheduler system , yes ! just one binary solve all the problems !)☆195Mar 21, 2026Updated 2 months ago
- 在内网中使用代理上互联网☆10Mar 7, 2023Updated 3 years ago
- Helm Chart for Samba4☆10Oct 13, 2021Updated 4 years ago
- The classic movies redux with machine learning using TensorFlow and Keras.☆11Feb 12, 2019Updated 7 years ago
- Dagger Quickstart - Example Application☆10May 21, 2026Updated 3 weeks ago
- 一款优秀的在线文 件预览解决方案，，使用主流springboot+maven搭建，支持doc、docx、ppt、pptx、xls、xlsx、zip、rar、mp4、mp3以及众多类文本如txt、html、xml、java、properties、sql、js、md、json、c…☆16Dec 6, 2022Updated 3 years ago
- 小蓝本(https://www.xiaolanben.com/) 爬虫的 h_sign 签名JSRPC实现。nodejs 补环境也实现了☆14Apr 30, 2024Updated 2 years ago
- HTML DOM Query Language for XGo☆42Feb 16, 2026Updated 4 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A simple sync tool to sync task from Workflowy to Teambition☆32Oct 4, 2017Updated 8 years ago
- Prediction model for Kaggle/Rossmann competition.☆13Nov 23, 2015Updated 10 years ago
- This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.☆1,226Nov 7, 2023Updated 2 years ago
- Python Web Development with Sanic☆10May 16, 2022Updated 4 years ago
- API rapid development framework for SANIC, Inspired by Django REST Framework.☆11Aug 29, 2025Updated 9 months ago
- Pyppeteer integration for Scrapy☆58Feb 26, 2021Updated 5 years ago
- Scrapy Pyppeteer Demo☆24Jul 13, 2018Updated 7 years ago
- Scrapy Universal Spider☆57Aug 26, 2017Updated 8 years ago
- Strong, Simple, and Precise, (and now async!) security for Sanic APIs☆14May 29, 2026Updated 2 weeks ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- 基于Azure OpenAI的飞书机器人☆13Apr 18, 2023Updated 3 years ago
- ☆11Dec 23, 2017Updated 8 years ago
- ☆11Mar 16, 2022Updated 4 years ago
- 一个纯 Go 实现的游戏手柄网络转发工具（实际上用的库并不纯）☆13May 8, 2020Updated 6 years ago
- ScrapyDemo : Redis MySQLdb logging IngoreHttpRequestMiddleware UserAgentMiddleware HttpProxyMiddleware rules☆38Jun 28, 2016Updated 9 years ago
- 基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star☆35Oct 25, 2019Updated 6 years ago
- A fast and small port of Zstandard to WASM. (Decompress-only for now).☆14Sep 24, 2023Updated 2 years ago