rmax / scrapy-inline-requests

Related projects: ⓘ

• scrapy-plugins / scrapy-magicfields
  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
  ☆56Updated 2 years ago
• llonchj / scrapy-sentry
  Sentry component for Scrapy
  ☆86Updated last year
• jschnurr / scrapyscript
  Run a Scrapy spider programmatically from a script or a Celery task - no project required.
  ☆121Updated 3 months ago
• scrapinghub / shub
  Scrapinghub Command Line Client
  ☆125Updated 4 months ago
• scrapy / queuelib
  Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
  ☆266Updated 4 months ago
• TeamHG-Memex / autopager
  Detect and classify pagination links
  ☆98Updated 4 years ago
• scrapy / itemadapter
  Common interface for data container classes
  ☆61Updated last month
• lopuhin / scrapy-pyppeteer
  Use pyppeteer from a Scrapy spider
  ☆60Updated 4 years ago
• scrapy-plugins / scrapy-deltafetch
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
  ☆264Updated 2 years ago
• scrapinghub / scrapy-poet
  Page Object pattern for Scrapy
  ☆119Updated 2 months ago
• scrapy / itemloaders
  Library to populate items using XPath and CSS with a convenient API
  ☆44Updated 3 months ago
• TeamHG-Memex / aquarium
  Splash + HAProxy + Docker Compose
  ☆196Updated 5 years ago
• scrapinghub / python-scrapinghub
  A client interface for Scrapinghub's API
  ☆202Updated 8 months ago
• scrapy / cssselect
  CSS Selectors for Python
  ☆291Updated 4 months ago
• scrapinghub / testspiders
  Useful test spiders for Scrapy
  ☆184Updated 4 years ago
• scrapinghub / scrapy-frontera
  More flexible and featured Frontera scheduler for Scrapy
  ☆35Updated 8 months ago
• scrapy-plugins / scrapy-jsonschema
  Scrapy schema validation pipeline and Item builder using JSON Schema
  ☆44Updated 3 years ago
• elacuesta / scrapy-pyppeteer
  Pyppeteer integration for Scrapy
  ☆60Updated 3 years ago
• roycehaynes / scrapy-rabbitmq
  A RabbitMQ Scheduler for Scrapy
  ☆85Updated 2 years ago
• jxltom / scrapymon
  Simple Web UI for Scrapy spider management via Scrapyd
  ☆49Updated 6 years ago
• scrapinghub / scrapy-autounit
  Automatic unit test generation for Scrapy.
  ☆55Updated 3 years ago
• djm / python-scrapyd-api
  A Python wrapper for working with Scrapyd's API.
  ☆267Updated last month
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated last year
• scrapinghub / arche
  Analyze scraped data
  ☆47Updated 4 years ago
• lorien / user_agent
  Generator of User-Agent header
  ☆335Updated 2 months ago
• dfdeshom / scrapy-kafka
  Kafka-based components for Scrapy
  ☆79Updated 6 years ago
• scrapinghub / js2xml
  Convert Javascript code to an XML document
  ☆186Updated 2 years ago
• rmax / scrapydo
  Crochet-based blocking API for Scrapy.
  ☆46Updated 7 years ago
• croqaz / awesome-scrapy
  🕶 Awesome list of Scrapy tools and libraries
  ☆54Updated 4 years ago
• scrapy / w3lib
  Python library of web-related functions
  ☆388Updated 3 months ago