Alternatives and similar repositories for scrapy-puppeteer

Users that are interested in scrapy-puppeteer are comparing it to the libraries listed below

• elacuesta / scrapy-pyppeteer

  View on GitHub
  Pyppeteer integration for Scrapy
  ☆58Feb 26, 2021Updated 4 years ago
• lopuhin / scrapy-pyppeteer

  View on GitHub
  Use pyppeteer from a Scrapy spider
  ☆59Feb 5, 2020Updated 6 years ago
• Gerapy / GerapyProxy

  View on GitHub
  A package for supporting proxy in Scrapy & Gerapy
  ☆11Jul 15, 2020Updated 5 years ago
• Gerapy / GerapyPyppeteer

  View on GitHub
  Downloader Middleware to support Pyppeteer in Scrapy & Gerapy
  ☆136Dec 27, 2021Updated 4 years ago
• clemfromspace / scrapy-selenium

  View on GitHub
  Scrapy middleware to handle javascript pages using selenium
  ☆958Jul 8, 2024Updated last year
• opeyemibami / TDD-in-MLOps

  View on GitHub
  This repo is an approach to TDD in machine learning model operation. it covers project structure, testing essentials using pytest with Gi…
  ☆15Dec 2, 2020Updated 5 years ago
• Germey / Blog

  View on GitHub
  My Personal Blog
  ☆13Updated this week
• Germey / JMeterMonitor

  View on GitHub
  JMeter Tester with Influxdb and Grafana
  ☆14Apr 10, 2020Updated 5 years ago
• Python3WebSpider / ScrapyPyppeteerDeprecated

  View on GitHub
  Scrapy Pyppeteer Demo
  ☆24Jul 13, 2018Updated 7 years ago
• stummjr / scrapy-fieldstats

  View on GitHub
  A Scrapy extension to log items coverage when the spider shuts down
  ☆19Apr 11, 2020Updated 5 years ago
• Gerapy / GerapyAutoExtractor

  View on GitHub
  Auto Extractor Module
  ☆334Aug 19, 2024Updated last year
• cdrx / scrapyd-authenticated

  View on GitHub
  Docker container running scrapyd with HTTP authentication
  ☆41May 14, 2024Updated last year
• scrapinghub / spidermon

  View on GitHub
  Scrapy Extension for monitoring spiders execution.
  ☆553Updated this week
• pitmonticone / RedditTextClassification

  View on GitHub
  Reddit Gender Text-Classification.
  ☆11Apr 16, 2023Updated 2 years ago
• luzihang123 / scrapyd_pyppeteer

  View on GitHub
  可以用于scrapydweb的scrapyd节点，使用pyppeteer，在scrapy中异步使用
  ☆12Dec 8, 2022Updated 3 years ago
• scrapinghub / web-poet

  View on GitHub
  Web scraping Page Objects core library
  ☆104Jan 27, 2026Updated 2 weeks ago
• EmperorRP / EnergySwap

  View on GitHub
  P2P Energy trading platform built on top of Ethereum and Chainlink
  ☆13Jun 10, 2023Updated 2 years ago
• ryanvin / scrapydweb-docker

  View on GitHub
  scrapydweb, dockerfile
  ☆13Feb 1, 2021Updated 5 years ago
• alexyorke / butter-fingers

  View on GitHub
  A python library to generate highly realistic typos (fuzz-testing)
  ☆13Mar 16, 2025Updated 11 months ago
• teamdatatonic / terraform-google-scheduled-vertex-pipelines

  View on GitHub
  A Terraform module for scheduling Vertex Pipeline runs using Google Cloud Scheduler
  ☆13Dec 7, 2022Updated 3 years ago
• TeamHG-Memex / scrapy-kafka-export

  View on GitHub
  Scrapy extension which writes crawled items to Kafka
  ☆30Updated this week
• ScrapingBee / scrapingbee-python

  View on GitHub
  A Python library to interact with ScrapingBee's API for headless browsers and proxy rotation
  ☆29Feb 5, 2026Updated last week
• TeamHG-Memex / arachnado

  View on GitHub
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆160Updated this week
• CooperMin / qichacha

  View on GitHub
  企查查企业分类信息采集
  ☆43Apr 2, 2020Updated 5 years ago
• scrapy-plugins / scrapy-zyte-smartproxy

  View on GitHub
  Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
  ☆368Mar 24, 2025Updated 10 months ago
• Python3WebSpider / Scrape

  View on GitHub
  Platform of Web Views to Scrape
  ☆11Jun 7, 2020Updated 5 years ago
• Python3WebSpider / ScrapyPyppeteer

  View on GitHub
  Scrapy Pyppeteer Demo
  ☆12Jul 30, 2020Updated 5 years ago
• scrapy-plugins / scrapy-headless

  View on GitHub
  ☆29Apr 28, 2021Updated 4 years ago
• scrapy-plugins / scrapy-splash

  View on GitHub
  Scrapy+Splash for JavaScript integration
  ☆3,242Feb 11, 2025Updated last year
• crawlaio / scrapy-redis-sentinel

  View on GitHub
  scrapy-redis-sentinel 基于 scrapy-redis 的基础上 新增 哨兵（sentinel）连接模式 以及 集群（cluster）连接模式。
  ☆30Mar 31, 2023Updated 2 years ago
• AaronJny / scrapy_redis_expiredupefilter

  View on GitHub
  scrapy-redis-expiredupefilter是基于scrapy-redis修改来的一款scrapy分布式爬虫框架，它支持为请求指纹设置生命周期，请求指纹生命周期结束后将在不影响其他指纹的情况下自动清除。
  ☆10Aug 6, 2019Updated 6 years ago
• Python3WebSpider / ScrapyRedisBloomFilter

  View on GitHub
  Scrapy Redis Bloom Filter
  ☆178Jul 25, 2021Updated 4 years ago
• dfdeshom / scrapy-kafka

  View on GitHub
  Kafka-based components for Scrapy
  ☆78Apr 10, 2018Updated 7 years ago
• GaryBoone / PresidentialMonteCarlo

  View on GitHub
  Presidential election Monte Carlo simulation in Go based on latest polling from Huffington Post API
  ☆30Oct 30, 2016Updated 9 years ago
• scrapedin / scrapedin

  View on GitHub
  Generate potential email addresses from LinkedIn
  ☆17Jun 18, 2021Updated 4 years ago
• harryanon / social_mapper

  View on GitHub
  ☆12Aug 9, 2018Updated 7 years ago
• ispras / scrapy-puppeteer-service

  View on GitHub
  A special service that runs puppeteer instances.
  ☆18Jan 29, 2026Updated 2 weeks ago
• rmax / scrapy-inline-requests

  View on GitHub
  A decorator to write coroutine-like spider callbacks.
  ☆109Dec 26, 2022Updated 3 years ago
• scrapy-plugins / scrapy-playwright

  View on GitHub
  🎭 Playwright integration for Scrapy
  ☆1,361Jan 21, 2026Updated 3 weeks ago