Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner
☆111May 11, 2018Updated 8 years ago
Alternatives and similar repositories for Distributed-Multi-User-Scrapy-System-with-a-Web-UI
Users that are interested in Distributed-Multi-User-Scrapy-System-with-a-Web-UI are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Simple Web UI for Scrapy spider management via Scrapyd☆50Jun 25, 2018Updated 8 years ago
- 爬虫管理平台☆31Dec 8, 2022Updated 3 years ago
- Creating Scrapy scrapers via the Django admin interface☆1,153Feb 19, 2022Updated 4 years ago
- scrapy-ui☆16Feb 21, 2014Updated 12 years ago
- Command line client for Scrapyd server☆771Feb 27, 2026Updated 8 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- A django admin site for scrapy☆45Nov 1, 2017Updated 9 years ago
- MongoDB pipeline for Scrapy. It allows to update existing entries (set new values or add elements to array) when item values are spread o…☆10Jan 6, 2013Updated 13 years ago
- Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI.…☆3,410Feb 19, 2025Updated last year
- Mission - scraping the planet, one website at a time☆10Dec 7, 2022Updated 3 years ago
- a demo of scrapy + selenium☆20Sep 18, 2019Updated 7 years ago
- Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls☆275Feb 26, 2025Updated last year
- admin ui for scrapy/open source scrapinghub☆59Apr 28, 2021Updated 5 years ago
- This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.☆1,223Nov 7, 2023Updated 2 years ago
- Using Multi-Module architecture and Reactive pattern to build Audio Book App for Android Platform☆15Aug 2, 2020Updated 6 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Search engine base (crawler, indexer and parser) using Python, Celery, RabbitMQ, CouchDB and Whoosh.☆10Jun 10, 2025Updated last year
- A simple money Pooling and Managing Dapp built on top of Ethereum Blockchain using DAI stablecoin.☆10Feb 14, 2026Updated 8 months ago
- A service daemon to run Scrapy spiders☆3,100Oct 23, 2026Updated last week
- A freelancing platform☆11Sep 8, 2021Updated 5 years ago
- HTTP API for Scrapy spiders☆885Oct 13, 2026Updated 3 weeks ago
- A .Net based Web Scraper using the WebBrowser control☆15Jan 13, 2021Updated 5 years ago
- Creates a pipeline Airflow and Scrapy to output an average image composition of everyone's face in a given website☆43Oct 13, 2017Updated 9 years ago
- Scrapy project boilerplate done right☆49Feb 26, 2026Updated 8 months ago
- A thread-safe Proxy Pool that automatically crawl free proxies.☆16Sep 7, 2017Updated 9 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Digital store on top of Django Oscar.☆19Nov 30, 2024Updated last year
- Tagged pages for django CMS 3☆13Updated this week
- A daemon for scheduling Scrapy spiders☆65May 28, 2021Updated 5 years ago
- A scalable frontier for web crawlers☆1,337Jun 6, 2025Updated last year
- Mastering-Scala-Machine-Learning☆37Jan 30, 2023Updated 3 years ago
- Scrapy spiders of major websites. Google Play Store, Facebook, Instagram, Ebay, YTS Movies, Amazon☆298Jul 20, 2017Updated 9 years ago
- Implement scrapy with asyncio☆71Jul 23, 2026Updated 3 months ago
- Python web visualize build on the awesome web framework sanic☆62Jul 12, 2017Updated 9 years ago
- A RabbitMQ Scheduler for Scrapy☆86Aug 9, 2022Updated 4 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- This small app help you delete your old twit☆13Jun 14, 2023Updated 3 years ago
- Random proxy middleware for Scrapy☆1,667Oct 1, 2019Updated 7 years ago
- A Python click type which is similar to click.Path but returns a Pathlib path☆13Sep 8, 2026Updated last month
- Scrapes upwork.com using BeautifulSoup and Selenium☆12Jul 5, 2017Updated 9 years ago
- A simple Django web app for searching the Paradise Papers dataset backed by Neo4j☆84Sep 20, 2024Updated 2 years ago
- MongoDB pipeline for Scrapy. This module supports both MongoDB in standalone setups and replica sets. scrapy-mongodb will insert the item…☆355Apr 6, 2021Updated 5 years ago
- 一款基于SQLAlchemy生成models和controller层的代码生成器；controller层实现了基于实体的简 单增删改查操作；☆14Dec 16, 2025Updated 10 months ago