Insutanto / scrapy-distributed

Related projects ⓘ

Alternatives and complementary repositories for scrapy-distributed

• my8100 / logparser
  A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
  ☆89Updated 2 years ago
• clemfromspace / scrapy-puppeteer
  Scrapy + Puppeteer
  ☆111Updated 3 years ago
• crawlab-team / webspot
  An intelligent web service to automatically detect web content and extract information from it.
  ☆84Updated last year
• ConlinH / aio-scrapy
  Implement scrapy with asyncio
  ☆58Updated last month
• tenlee2012 / scrapy-kafka-redis
  Distributed crawling/scraping, Kafka And Redis based components for Scrapy
  ☆46Updated 4 years ago
• GeneralNewsExtractor / GneList
  A chrome extension to get XPath of list items in webpage easily.
  ☆35Updated 2 years ago
• roycehaynes / scrapy-rabbitmq
  A RabbitMQ Scheduler for Scrapy
  ☆85Updated 2 years ago
• aox-lei / scrapy-rabbitmq-scheduler
  rabbitmq的scrapy分布式爬虫
  ☆34Updated 3 years ago
• Python3WebSpider / ScrapyRedisBloomFilter
  Scrapy Redis Bloom Filter
  ☆175Updated 3 years ago
• cxapython / discogs_aio_spider
  基于httpx的一个大型项目 ，爬取黑胶唱片网站 Discogs
  ☆101Updated last year
• kingname / Tinepeas
  Tinepeas，我们自己的爬虫框架。
  ☆62Updated 3 months ago
• abo123456789 / leek
  Distributed task redisqueue(最简单python分布式函数调度框架)
  ☆63Updated last year
• Gerapy / GerapyPyppeteer
  Downloader Middleware to support Pyppeteer in Scrapy & Gerapy
  ☆137Updated 2 years ago
• elacuesta / scrapy-pyppeteer
  Pyppeteer integration for Scrapy
  ☆60Updated 3 years ago
• Gerapy / GerapyPlaywright
  Downloader Middleware to support Playwright in Scrapy & Gerapy
  ☆106Updated 2 years ago
• RedisBloom / redisbloom-py
  Python client for Redisbloom
  ☆76Updated last year
• Gerapy / GerapyAutoExtractor
  Auto Extractor Module
  ☆320Updated 3 months ago
• Boris-code / feaplat
  爬虫管理系统，支持集群，弹性伸缩。支持运行feapder、scrapy、selenium、playwright等各种框架及脚本
  ☆104Updated 7 months ago
• NightTeam / cellular-network-proxy_docker
  蜂窝网络代理服务器搭建DEMO-Docker版搭建方式
  ☆60Updated 5 years ago
• leffss / ScrapyRedisBloomFilterBlockCluster
  Scrapy Redis with Bloom Filter，support redis sentinel and cluster
  ☆23Updated last year
• tieyongjie / scrapy-fingerprint
  ☆56Updated last year
• crawlab-team / crawlab-core
  Backend core modules for Crawlab
  ☆49Updated 5 months ago
• crawlab-team / crawlab-sdk
  SDK for Crawlab, including SDK for different programming languages such as Python, Node.js and Java, and a CLI Tool written in Python.
  ☆55Updated 5 months ago
• use-py / usepy
  一个简单方便的Python工具包
  ☆18Updated last month
• scrapinghub / scrapy-frontera
  More flexible and featured Frontera scheduler for Scrapy
  ☆35Updated 10 months ago
• xiaxichen / zh_login
  知乎登录
  ☆22Updated 5 years ago
• bytebuff / aioScrapy
  基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star
  ☆35Updated 5 years ago
• synodriver / awsome_scrapy_utils
  awsome scrapy utils
  ☆54Updated 7 months ago
• Gerapy / GerapySelenium
  Downloader Middleware to support Selenium in Scrapy & Gerapy
  ☆31Updated 4 years ago
• ClericPy / ichrome
  Chrome controller for Humans, based on Chrome Devtools Protocol(CDP) and python3.7+.
  ☆236Updated 3 weeks ago