Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.
☆438Dec 30, 2022Updated 3 years ago
Alternatives and similar repositories for Crawling-Infrastructure
Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Javascript scraping module based on puppeteer for many different search engines...☆570Dec 30, 2022Updated 3 years ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆70Jun 8, 2021Updated 5 years ago
- 🧱 A uniform template to use as a foundation for Puppeteer bot construction.☆68May 6, 2021Updated 5 years ago
- 🛡🎭 A conceptual patch which modifies some vanilla puppeteer files to decrease detection rates.☆56Mar 6, 2021Updated 5 years ago
- ☆12May 7, 2023Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Scrapoxy has been discontinued.☆2,406Feb 7, 2026Updated 8 months ago
- A simple example of using Puppeteer to test your analytics setup☆14Aug 5, 2022Updated 4 years ago
- 📡 Renew the IP address of a tethered Android device via Node asynchronously.☆77Aug 3, 2023Updated 3 years ago
- Streaming web crawler with WebSocket API☆48May 15, 2026Updated 5 months ago
- POC code to crash Windows Event Logger Service☆27Oct 16, 2020Updated 6 years ago
- Passive TCP/IP Fingerprinting Tool. Run this on your server and find out what Operating Systems your clients are *really* using.☆451Sep 21, 2026Updated last month
- ☆174Dec 30, 2022Updated 3 years ago
- Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprint…☆5,152Jul 27, 2026Updated 3 months ago
- Solution to stop sites from fingerprinting your puppeteer☆131Jun 12, 2026Updated 4 months ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- A test suite of common scraper detection techniques. See how detectable your scraper stack is.☆138Oct 31, 2022Updated 3 years ago
- ☆20Apr 21, 2020Updated 6 years ago
- Puppeteer Pool, run a cluster of instances in parallel☆3,515Mar 1, 2026Updated 7 months ago
- Repo for hosting various scripts for creating users for password spraying and other password attacks.☆11Jul 9, 2020Updated 6 years ago
- Microsoft Applocker evasion tool☆39Nov 26, 2019Updated 6 years ago
- Additional module to use with 'puppeteer' for setting proxies per page basis.☆450Jun 9, 2024Updated 2 years ago
- The BlogDB Webservice☆13Feb 1, 2022Updated 4 years ago
- Distributed crawler powered by Headless Chrome☆5,632Apr 29, 2023Updated 3 years ago
- ☆10Dec 18, 2018Updated 7 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- Event Data Collector☆40Oct 21, 2026Updated last week
- #️⃣ 🕸️ 👤 HTTP Headers Hashing☆11Aug 27, 2023Updated 3 years ago
- A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…☆382Dec 30, 2022Updated 3 years ago
- Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.☆657Jun 5, 2021Updated 5 years ago
- Nodejs lib to parse Google SERP html pages☆48Jul 27, 2023Updated 3 years ago
- This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.☆1,222Nov 7, 2023Updated 2 years ago
- BH Cypher Queries picked up from random places☆42Dec 12, 2018Updated 7 years ago
- Proxies Puppeteer Page requests.☆214Aug 28, 2024Updated 2 years ago
- A simple, quick, and dirty websocket shell for PowerShell.☆20Jun 5, 2017Updated 9 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Article extraction benchmark: dataset and evaluation scripts☆378May 29, 2026Updated 5 months ago
- 🕵♂ Bot detection tests for Puppeteer. Hide and seek!☆105Mar 5, 2023Updated 3 years ago
- Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data …☆26,288Updated this week
- A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based componen…☆61May 10, 2026Updated 5 months ago
- C&C to deliver files and shuttle command execution instructions between an external actor and an internal agent with the help of Firefox …☆38Dec 8, 2022Updated 3 years ago
- 👀 Analyze Websites and Resources They Request☆24Feb 3, 2019Updated 7 years ago
- Serverless fax gateway for reliable, high-throughput faxing☆11Mar 9, 2023Updated 3 years ago