rmax/scrapydo external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

rmax/scrapydo

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/rmax/scrapydo)

Close

rmax / scrapydoView on GitHubMore

• External links
• Readme badge

Crochet-based blocking API for Scrapy.

☆46Feb 24, 2017Updated 9 years ago

Alternatives and similar repositories for scrapydo

Users that are interested in scrapydo are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• rmax / databrewer

  View on GitHub
  The missing datasets manager. Like hombrew but for datasets. CLI-tool for search and discover datasets!
  ☆41May 29, 2017Updated 8 years ago
• stummjr / scrapy-fieldstats

  View on GitHub
  A Scrapy extension to log items coverage when the spider shuts down
  ☆19Apr 11, 2020Updated 5 years ago
• scrapinghub / page_finder

  View on GitHub
  Find which links on a web page are pagination links
  ☆29Jan 12, 2017Updated 9 years ago
• itamarst / crochet

  View on GitHub
  Crochet: use Twisted anywhere!
  ☆240Sep 3, 2024Updated last year
• ArturGaspar / scrapy-qtwebkit

  View on GitHub
  ☆13Dec 4, 2019Updated 6 years ago
• zytedata / html-text

  View on GitHub
  ☆18Oct 6, 2025Updated 5 months ago
• inspirehep / hepcrawl

  View on GitHub
  Scrapy project for feeds into INSPIRE-HEP
  ☆19Sep 4, 2025Updated 6 months ago
• TeamHG-Memex / tor-proxy

  View on GitHub
  a tor socks proxy docker image
  ☆12Feb 10, 2026Updated last month
• scrapinghub / aile

  View on GitHub
  Automatic Item List Extraction
  ☆86Jun 15, 2016Updated 9 years ago
• scrapinghub / autopager

  View on GitHub
  Detect and classify pagination links
  ☆15Sep 9, 2020Updated 5 years ago
• TeamHG-Memex / arachnado

  View on GitHub
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆161Feb 10, 2026Updated last month
• nyov / scrapyext

  View on GitHub
  scrapy-extras -- a collection of code samples and modules for the Scrapy framework.
  ☆14Dec 14, 2020Updated 5 years ago
• TeamHG-Memex / MaybeDont

  View on GitHub
  A component that tries to avoid downloading duplicate content
  ☆28Feb 10, 2026Updated last month
• TeamHG-Memex / scrapy-dockerhub

  View on GitHub
  [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
  ☆12Feb 23, 2026Updated last month
• redapple / parslepy

  View on GitHub
  Python implementation of the Parsley language for extracting structured data from web pages
  ☆92Oct 26, 2017Updated 8 years ago
• kserhii / money-parser

  View on GitHub
  Price and currency parsing utility
  ☆27Mar 6, 2023Updated 3 years ago
• rmax / scrapy-boilerplate

  View on GitHub
  Small set of utilities to simplify writing Scrapy spiders.
  ☆50Jul 24, 2015Updated 10 years ago
• TeamHG-Memex / extract-html-diff

  View on GitHub
  extract difference between two html pages
  ☆33Feb 10, 2026Updated last month
• scrapinghub / scrapyrt

  View on GitHub
  HTTP API for Scrapy spiders
  ☆881Feb 16, 2026Updated last month
• wlabatey / job_scraper

  View on GitHub
  A job scraper using the Scrapy framework
  ☆16Oct 20, 2017Updated 8 years ago
• scrapinghub / product-extraction-benchmark

  View on GitHub
  ☆16Apr 24, 2024Updated last year
• alertot / detectem

  View on GitHub
  detectem - detect software and its version on websites.
  ☆157Mar 25, 2021Updated 4 years ago
• stav / scrapybox

  View on GitHub
  Scrapy GUI
  ☆12Feb 26, 2021Updated 5 years ago
• scrapy / scrapy-lint

  View on GitHub
  A linter for Scrapy projects.
  ☆21Feb 25, 2026Updated 3 weeks ago
• elacuesta / scrapy-pyppeteer

  View on GitHub
  Pyppeteer integration for Scrapy
  ☆58Feb 26, 2021Updated 5 years ago
• seagatesoft / webdext

  View on GitHub
  Intelligent Web Data Extractor
  ☆74Dec 5, 2022Updated 3 years ago
• TeamHG-Memex / soft404

  View on GitHub
  A classifier for detecting soft 404 pages
  ☆60Feb 10, 2026Updated last month
• scrapinghub / scrapy-poet

  View on GitHub
  Page Object pattern for Scrapy
  ☆127Mar 15, 2026Updated last week
• commonsearch / gumbocy

  View on GitHub
  Python binding for gumbo-parser using Cython
  ☆14Aug 16, 2016Updated 9 years ago
• TeamHG-Memex / undercrawler

  View on GitHub
  A generic crawler
  ☆79Feb 10, 2026Updated last month
• scrapinghub / kafka-scanner

  View on GitHub
  High Level Kafka Scanner
  ☆19Sep 29, 2017Updated 8 years ago
• blaze / datafabric

  View on GitHub
  A distributed in-memory fabric based on shared-memory blocks and datashape. Any language can operate on the data.
  ☆13Feb 12, 2016Updated 10 years ago
• scrapinghub / skinfer

  View on GitHub
  Skinfer is a tool for inferring and merging JSON schemas
  ☆141Apr 24, 2024Updated last year
• scrapinghub / js2xml

  View on GitHub
  Convert Javascript code to an XML document
  ☆187Mar 14, 2022Updated 4 years ago
• haizi-zh / scrapy-qiniu

  View on GitHub
  Scrapy中，将网络资源（文件、图像等）存储在七牛上的Pipeline扩展
  ☆24Dec 26, 2015Updated 10 years ago
• numercfd / aws-fasi

  View on GitHub
  Failover AWS Spot Instances
  ☆11Dec 8, 2017Updated 8 years ago
• scrapinghub / flatson

  View on GitHub
  Tool to flatten stream of JSON-like objects, configured via schema
  ☆33Oct 19, 2019Updated 6 years ago
• commonsearch / urlparse4

  View on GitHub
  Faster replacement for Python's urlparse module
  ☆46Sep 30, 2018Updated 7 years ago
• TeamHG-Memex / autopager

  View on GitHub
  Detect and classify pagination links
  ☆107Feb 10, 2026Updated last month