LiuXingMing/Scrapy_Redis_Bloomfilter external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

LiuXingMing/Scrapy_Redis_Bloomfilter

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/LiuXingMing/Scrapy_Redis_Bloomfilter)

Close

LiuXingMing / Scrapy_Redis_BloomfilterView on GitHubMore

• External links
• Readme badge

基于Redis的Bloomfilter去重，并将其扩展到Scrapy框架。

☆345Feb 26, 2023Updated 3 years ago

Alternatives and similar repositories for Scrapy_Redis_Bloomfilter

Users that are interested in Scrapy_Redis_Bloomfilter are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• Python3WebSpider / ScrapyRedisBloomFilter

  View on GitHub
  Scrapy Redis Bloom Filter
  ☆173Jul 25, 2021Updated 5 years ago
• rmax / scrapy-redis

  View on GitHub
  Redis-based components for Scrapy.
  ☆5,647Updated this week
• gnemoug / distribute_crawler

  View on GitHub
  使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
  ☆3,234Apr 18, 2017Updated 9 years ago
• LiuXingMing / SinaSpider

  View on GitHub
  新浪微博爬虫（Scrapy、Redis）
  ☆3,283Sep 5, 2018Updated 8 years ago
• DormyMo / SpiderKeeper

  View on GitHub
  admin ui for scrapy/open source scrapinghub
  ☆2,766May 4, 2023Updated 3 years ago
• Deploy to Railway using AI coding agents - Free Credits Offer • Ad
  Use Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
• luyishisi / Anti-Anti-Spider

  View on GitHub
  越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
  ☆7,293Oct 17, 2021Updated 5 years ago
• Gerapy / Gerapy

  View on GitHub
  Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
  ☆3,515Jul 4, 2026Updated 3 months ago
• xsren / frontera-docs-zh_CN

  View on GitHub
  frontera的中文翻译文档
  ☆35Mar 10, 2018Updated 8 years ago
• shisiying / tc_zufang

  View on GitHub
  使用scrapy,redis, mongodb,django实现的一个分布式网络爬虫,底层存储mongodb,分布式使用redis实现,使用django可视化爬虫
  ☆276May 1, 2018Updated 8 years ago
• KDF5000 / RSpider

  View on GitHub
  一个基于scrapy-redis的分布式爬虫模板
  ☆43Jul 4, 2017Updated 9 years ago
• aivarsk / scrapy-proxies

  View on GitHub
  Random proxy middleware for Scrapy
  ☆1,667Oct 1, 2019Updated 7 years ago
• kongtianyi / BloomFilterRedis

  View on GitHub
  Bloom filter based on redis.
  ☆46Mar 16, 2023Updated 3 years ago
• qiyeboy / IPProxyPool

  View on GitHub
  IPProxyPool代理池项目，提供代理ip
  ☆4,288Jul 13, 2018Updated 8 years ago
• scrapy-plugins / scrapy-deltafetch

  View on GitHub
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
  ☆275Feb 26, 2025Updated last year
• End-to-end encrypted cloud storage - Proton Drive • Ad
  Special offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
• istresearch / scrapy-cluster

  View on GitHub
  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
  ☆1,222Nov 7, 2023Updated 2 years ago
• leffss / ScrapyRedisBloomFilterBlockCluster

  View on GitHub
  Scrapy Redis with Bloom Filter，support redis sentinel and cluster
  ☆25Mar 31, 2023Updated 3 years ago
• my8100 / scrapydweb

  View on GitHub
  Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI.…
  ☆3,410Feb 19, 2025Updated last year
• ramsayleung / jd_spider

  View on GitHub
  Two dumb distributed crawlers
  ☆718Apr 8, 2019Updated 7 years ago
• Johnson0722 / News_scrapy_redis

  View on GitHub
  ☆30Jul 5, 2018Updated 8 years ago
• AlexTan-b-z / ZhihuSpider

  View on GitHub
  知乎分布式爬虫（Scrapy、Redis）
  ☆168Feb 18, 2018Updated 8 years ago
• kohn / HttpProxyMiddleware

  View on GitHub
  A middleware for scrapy. Used to change HTTP proxy from time to time.
  ☆323Feb 1, 2018Updated 8 years ago
• younghz / sr-chn

  View on GitHub
  scrapy-redis代码研究
  ☆14Oct 10, 2014Updated 12 years ago
• MollyMmm / anyproxy_weixin

  View on GitHub
  使用anyproxy获取wx_gzh文章
  ☆11Apr 18, 2018Updated 8 years ago
• Managed Database hosting by DigitalOcean • Ad
  PostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
• scrapy-plugins / scrapy-splash

  View on GitHub
  Scrapy+Splash for JavaScript integration
  ☆3,225Feb 11, 2025Updated last year
• SpiderClub / haipproxy

  View on GitHub
  High available distributed ip proxy pool, powerd by Scrapy and Redis
  ☆5,525Dec 26, 2022Updated 3 years ago
• inlike / CookiePool

  View on GitHub
  一个强大的Cookie池项目，融合scrapy/requests/chrome储存cookie/cookie字符串/selenium等cookie形式
  ☆232Mar 13, 2020Updated 6 years ago
• leng-yue / Lengyue-Vcode

  View on GitHub
  Project Stopped
  ☆1,049Dec 24, 2019Updated 6 years ago
• jaybaird / python-bloomfilter

  View on GitHub
  Scalable Bloom Filter implemented in Python
  ☆1,619Jul 1, 2021Updated 5 years ago
• LiuXingMing / Tmall1212

  View on GitHub
  天猫双12爬虫，附商品数据。
  ☆203Dec 12, 2016Updated 9 years ago
• flisky / scrapy-phantomjs-downloader

  View on GitHub
  PhantomJS Downloader for Scrapy, Yeah!
  ☆93Aug 11, 2014Updated 12 years ago
• crawlaio / scrapy-redis-sentinel

  View on GitHub
  scrapy-redis-sentinel 基于 scrapy-redis 的基础上 新增 哨兵（sentinel）连接模式 以及 集群（cluster）连接模式。
  ☆30Mar 31, 2023Updated 3 years ago
• jhao104 / proxy_pool

  View on GitHub
  Python ProxyPool for web spider
  ☆23,778Jun 15, 2026Updated 4 months ago
• Deploy on Railway without the complexity - Free Credits Offer • Ad
  Connect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
• shikanon / proxy_scrapy

  View on GitHub
  proxy_scrapy是一个scrapy搭建的代理模块，主要包括代理抓取、代理测试和使用代理三个模块。包括了对主要的代理网站的抓取和代理稳定性的测试，并整合进scrapy爬虫当中。
  ☆10Jan 20, 2017Updated 9 years ago
• scrapinghub / frontera

  View on GitHub
  A scalable frontier for web crawlers
  ☆1,337Jun 6, 2025Updated last year
• HatBoy / BloomFilter

  View on GitHub
  Python+Redis实现的BloomFilter
  ☆15Nov 27, 2016Updated 9 years ago
• use-py / use-notify

  View on GitHub
  一个简易且可扩展的消息通知组件，已内置钉钉，企微，bark，pushover，pushdeer，飞书，邮件等消息通知组件
  ☆27Jul 11, 2026Updated 3 months ago
• Insutanto / scrapy-distributed

  View on GitHub
  A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based componen…
  ☆61May 10, 2026Updated 5 months ago
• Python3WebSpider / CookiesPool

  View on GitHub
  Cookies Pool
  ☆569Jun 9, 2020Updated 6 years ago
• chyroc / WechatSogou

  View on GitHub
  基于搜狗微信搜索的微信公众号爬虫接口
  ☆6,407Oct 23, 2026Updated last week