tenlee2012/scrapy-kafka-redis external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

tenlee2012/scrapy-kafka-redis

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/tenlee2012/scrapy-kafka-redis)

Close

tenlee2012 / scrapy-kafka-redisView on GitHubMore

• External links
• Readme badge

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

☆45Nov 13, 2020Updated 5 years ago

Alternatives and similar repositories for scrapy-kafka-redis

Users that are interested in scrapy-kafka-redis are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• dfdeshom / scrapy-kafka

  View on GitHub
  Kafka-based components for Scrapy
  ☆78Apr 10, 2018Updated 8 years ago
• AaronJny / scrapy_redis_expiredupefilter

  View on GitHub
  scrapy-redis-expiredupefilter是基于scrapy-redis修改来的一款scrapy分布式爬虫框架，它支持为请求指纹设置生命周期，请求指纹生命周期结束后将在不影响其他指纹的情况下自动清除。
  ☆10Aug 6, 2019Updated 7 years ago
• sunhailin-Leo / Scrapy-Kafka-Demo

  View on GitHub
  Scrapy and Kafka
  ☆14Feb 7, 2018Updated 8 years ago
• OneJane / OneJane.github.io

  View on GitHub
  个人博客
  ☆13Feb 2, 2023Updated 3 years ago
• arkady-emelyanov / kafkabeat

  View on GitHub
  Elastic Beat that consumes Apache Kafka topics and ships events to Elasticsearch, Logstash or any Beats output. Plain and JSON codecs.
  ☆12Apr 29, 2019Updated 7 years ago
• Deploy to Railway using AI coding agents - Free Credits Offer • Ad
  Use Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
• roycehaynes / scrapy-rabbitmq

  View on GitHub
  A RabbitMQ Scheduler for Scrapy
  ☆86Aug 9, 2022Updated 4 years ago
• samuelyao314 / nsqbook

  View on GitHub
  nsq官方文档翻译和源码学习
  ☆11Sep 10, 2014Updated 12 years ago
• WXjzcccc / easyFrida

  View on GitHub
  难用的frida通用脚本工具
  ☆48Jul 31, 2025Updated last year
• WXjzcccc / recoverMnemonic

  View on GitHub
  以太坊钱包助记词顺序恢复工具
  ☆16Aug 4, 2025Updated last year
• Masutangu / SuperScripter

  View on GitHub
  A Simple Tool to Distribute/Administrate Your Scripts
  ☆12Sep 10, 2015Updated 11 years ago
• auroraruanjian / go_mouyin

  View on GitHub
  使用Golang wails为GUI框架编写的某音客户端，集成登录，搜索，多线程数据抓取
  ☆12May 8, 2023Updated 3 years ago
• Liangchengdeye / Requests_Html_Spider

  View on GitHub
  requests升级版requests-html 爬虫编写及通用爬虫模块搭建
  ☆11Nov 21, 2018Updated 7 years ago
• Germey / AdslProxy

  View on GitHub
  ☆17Jul 14, 2017Updated 9 years ago
• LiuXingMing / Scrapy_Redis_Bloomfilter

  View on GitHub
  基于Redis的Bloomfilter去重，并将其扩展到Scrapy框架。
  ☆345Feb 26, 2023Updated 3 years ago
• 1-Click AI Models by DigitalOcean Gradient • Ad
  Deploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
• huichen / batchqueue

  View on GitHub
  批处理延迟任务队列
  ☆53Aug 8, 2013Updated 13 years ago
• michellebonat / tf_text_classify

  View on GitHub
  The classic movies redux with machine learning using TensorFlow and Keras.
  ☆11Feb 12, 2019Updated 7 years ago
• Gerapy / GerapyAutoExtractor

  View on GitHub
  Auto Extractor Module
  ☆337Aug 19, 2024Updated 2 years ago
• BruceDone / clock

  View on GitHub
  可视化任务调度系统，精简到一个二进制文件 (Web visual task scheduler system , yes ! just one binary solve all the problems !)
  ☆196Mar 21, 2026Updated 6 months ago
• JeffreyDing11223 / kago

  View on GitHub
  the Go client to use Kafka . Based on sarama and sarama-cluster.
  ☆14Aug 10, 2018Updated 8 years ago
• AceDataCloud / WeChatClaudeCode

  View on GitHub
  Use WeChat to connect Claude Code
  ☆26Apr 12, 2026Updated 6 months ago
• CrazyLittleArmy / DocumentPreview

  View on GitHub
  一款优秀的在线文件预览解决方案，，使用主流springboot+maven搭建，支持doc、docx、ppt、pptx、xls、xlsx、zip、rar、mp4、mp3以及众多类文本如txt、html、xml、java、properties、sql、js、md、json、c…
  ☆16Dec 6, 2022Updated 3 years ago
• dagger / hello-dagger

  View on GitHub
  Dagger Quickstart - Example Application
  ☆11Jun 26, 2026Updated 3 months ago
• trganda / wechat_article_spider

  View on GitHub
  微信公众号文章爬取
  ☆13Apr 27, 2023Updated 3 years ago
• Proton VPN Special Offer - Get 70% off • Ad
  Special partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
• r1is / xiaolanben_h_sign

  View on GitHub
  小蓝本(https://www.xiaolanben.com/) 爬虫的 h_sign 签名JSRPC实现。nodejs 补环境也实现了
  ☆13Apr 30, 2024Updated 2 years ago
• kingname / TeamFlowy

  View on GitHub
  A simple sync tool to sync task from Workflowy to Teambition
  ☆32Oct 4, 2017Updated 9 years ago
• zkqiang / job-spider

  View on GitHub
  多线程爬取互联网行业常用招聘网站
  ☆29Mar 4, 2018Updated 8 years ago
• xiaxichen / zh_login

  View on GitHub
  知乎登录
  ☆22Mar 18, 2019Updated 7 years ago
• istresearch / scrapy-cluster

  View on GitHub
  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
  ☆1,222Nov 7, 2023Updated 2 years ago
• elacuesta / scrapy-pyppeteer

  View on GitHub
  Pyppeteer integration for Scrapy
  ☆58Feb 26, 2021Updated 5 years ago
• Tioit-Wang / sanic-rest-framework

  View on GitHub
  API rapid development framework for SANIC, Inspired by Django REST Framework.
  ☆12Aug 29, 2025Updated last year
• Python3WebSpider / ScrapyUniversal

  View on GitHub
  Scrapy Universal Spider
  ☆57Aug 26, 2017Updated 9 years ago
• GeneralNewsExtractor / GneList

  View on GitHub
  A chrome extension to get XPath of list items in webpage easily.
  ☆34Mar 11, 2022Updated 4 years ago
• Wordpress hosting with auto-scaling - Free Trial Offer • Ad
  Fully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
• MarkHoo / django-xadmin

  View on GitHub
  ☆10Nov 1, 2018Updated 7 years ago
• Python3WebSpider / ScrapyPyppeteerDeprecated

  View on GitHub
  Scrapy Pyppeteer Demo
  ☆24Jul 13, 2018Updated 8 years ago
• siseng / siseng.github.io

  View on GitHub
  homepage
  ☆10Feb 15, 2023Updated 3 years ago
• mohuishou / Go-000

  View on GitHub
  ☆28Apr 20, 2021Updated 5 years ago
• fengxiaochuang / ScrapyDemo

  View on GitHub
  ScrapyDemo : Redis MySQLdb logging IngoreHttpRequestMiddleware UserAgentMiddleware HttpProxyMiddleware rules
  ☆37Jun 28, 2016Updated 10 years ago
• bytebuff / aioScrapy

  View on GitHub
  基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star
  ☆35Oct 25, 2019Updated 6 years ago
• HanEightTurtle / mitm_server_ql

  View on GitHub
  ☆11Mar 16, 2022Updated 4 years ago