canyousayyes / scrapy-web-crawler-by-rest-api
Example Scrapy project to crawl the web using the site's REST API
☆14Updated 6 years ago
Alternatives and similar repositories for scrapy-web-crawler-by-rest-api:
Users that are interested in scrapy-web-crawler-by-rest-api are comparing it to the libraries listed below
- Scrape websites asynchronously with Python 3.8+, Asyncio, & arsenic (aka Selenium for Async).☆57Updated 4 years ago
- Simple Web UI for Scrapy spider management via Scrapyd☆51Updated 6 years ago
- docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".☆42Updated 10 years ago
- Want to handle 100,000 messages in 90 seconds? Celery and Kombu are that awesome - Multiple publisher-subscriber demos for processing jso…☆41Updated 6 years ago
- A decorator to write coroutine-like spider callbacks.☆109Updated 2 years ago
- ☆29Updated 3 years ago
- Highly scalable webcrawler for towardsdatascience.com by using Python, Selenium, Docker, Kubernetes and the infrastructure of the Google …☆25Updated 3 years ago
- A Django GraphQL API example with Django Graphene☆23Updated 7 years ago
- A Django project to manage the Open Bank Project API via API Calls☆67Updated last week
- Scrapy + Puppeteer☆110Updated 3 years ago
- Software stack with latest Scrapy and updated deps☆64Updated 2 months ago
- Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner☆113Updated 6 years ago
- Zyte Automatic Extraction integration for Scrapy☆56Updated 3 years ago
- A helpful boilerplate for SaaS application w/ multi-tenant and JWT Authentication(Auth0) using DRF(Django REST Framework)☆13Updated 2 years ago
- Scrapy schema validation pipeline and Item builder using JSON Schema☆44Updated 4 years ago
- A scrapy extension to store requests and responses information in storage service☆26Updated 3 years ago
- Learn how to use websockets from scratch by creating a micro version of Jupyter. We'll be using both Python and JavaScript to make this h…☆21Updated 4 years ago
- Scrapy integration with Tor for anonymous web scraping☆46Updated 9 years ago
- Flask Boilerplate - Built with Automation Tools | AppSeed App Generator☆18Updated 3 years ago
- Scraping tweets quickly using celery, RabbitMQ and Docker cluster☆48Updated 2 years ago
- A scrapy middleware to use rotated proxy ip list.☆24Updated 6 years ago
- An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site☆127Updated 6 years ago
- Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.☆56Updated 3 years ago
- More flexible and featured Frontera scheduler for Scrapy☆36Updated 4 months ago
- Creates a pipeline Airflow and Scrapy to output an average image composition of everyone's face in a given website☆44Updated 7 years ago
- Lightweight library that converts a HTML webpage to JSON data using a template defined in JSON.☆21Updated 4 years ago
- A Scrapy pipeline module to persist items to a postgres table automatically.☆21Updated 7 years ago
- Reusable Django Application for Affiliate Programs and Campaign tracking, abandoned & unfinished☆23Updated 3 years ago
- Example of how to handle background processes with Falcon, Celery, and Docker☆35Updated 2 years ago
- Full stack, modern web application generator. Using Flask, Couchbase as database, Docker, Swagger, automatic HTTPS and more.☆59Updated 2 years ago