Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.
☆440Dec 30, 2022Updated 3 years ago
Alternatives and similar repositories for Crawling-Infrastructure
Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Cloud crawler functions for scrapeulous☆44Feb 24, 2021Updated 5 years ago
- Javascript scraping module based on puppeteer for many different search engines...☆570Dec 30, 2022Updated 3 years ago
- Minimal set of tools to conduct stealthy scraping.☆166Apr 21, 2023Updated 3 years ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆69Jun 8, 2021Updated 5 years ago
- A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.☆2,900Jul 3, 2021Updated 5 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- 🧱 A uniform template to use as a foundation for Puppeteer bot construction.☆68May 6, 2021Updated 5 years ago
- ☆12May 7, 2023Updated 3 years ago
- IP address API by ipapi.is: who owns an IP, where it is, and whether it is a datacenter, VPN, proxy, Tor exit or abuser. 1,000 free reque…☆119Updated this week
- Scrapoxy has been discontinued.☆2,402Feb 7, 2026Updated 9 months ago
- 💯 Teach puppeteer new tricks through plugins.☆7,425Jul 18, 2024Updated 2 years ago
- Passive TCP/IP Fingerprinting Tool. Run this on your server and find out what Operating Systems your clients are *really* using.☆451Sep 21, 2026Updated last month
- The best remote logging tool for iOS and Android apps☆22Sep 28, 2025Updated last year
- In-Memory Key-Value Database with Persistent File Storage☆16Sep 24, 2022Updated 4 years ago
- 📡 Renew the IP address of a tethered Android device via Node asynchronously.☆77Aug 3, 2023Updated 3 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- POC code to crash Windows Event Logger Service☆27Oct 16, 2020Updated 6 years ago
- A simple example of using Puppeteer to test your analytics setup☆14Aug 5, 2022Updated 4 years ago
- Streaming web crawler with WebSocket API☆48May 15, 2026Updated 6 months ago
- Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprint…☆5,167Jul 27, 2026Updated 3 months ago
- The web scraper that's nearly impossible to block - now called @ulixee/hero☆730Mar 7, 2023Updated 3 years ago
- ☆174Dec 30, 2022Updated 3 years ago
- Solution to stop sites from fingerprinting your puppeteer☆132Jun 12, 2026Updated 5 months ago
- A test suite of common scraper detection techniques. See how detectable your scraper stack is.☆138Oct 31, 2022Updated 4 years ago
- ☆20Apr 21, 2020Updated 6 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Search google, bing, yahoo, and other search engines with python☆678Apr 2, 2025Updated last year
- Distributed crawler powered by Headless Chrome☆5,632Apr 29, 2023Updated 3 years ago
- Repo for hosting various scripts for creating users for password spraying and other password attacks.☆11Jul 9, 2020Updated 6 years ago
- Puppeteer Pool, run a cluster of instances in parallel☆3,517Mar 1, 2026Updated 8 months ago
- Microsoft Applocker evasion tool☆39Nov 26, 2019Updated 6 years ago
- Additional module to use with 'puppeteer' for setting proxies per page basis.☆449Jun 9, 2024Updated 2 years ago
- Update of uncaptcha2 from 2019☆165Jan 5, 2021Updated 5 years ago
- C# application that allows you to quick run SSH commands against a host or list of hosts☆42Sep 21, 2020Updated 6 years ago
- ☆10Dec 18, 2018Updated 7 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- #️⃣ 🕸️ 👤 HTTP Headers Hashing☆11Aug 27, 2023Updated 3 years ago
- List of free and checked http, https, socks4 and socks5 proxies☆22Updated this week
- Event Data Collector☆40Oct 21, 2026Updated 3 weeks ago
- Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.☆656Jun 5, 2021Updated 5 years ago
- Web application/technology detection tool☆216Sep 1, 2023Updated 3 years ago
- This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.☆1,223Nov 7, 2023Updated 3 years ago
- Chromium Binary for AWS Lambda and Google Cloud Functions☆3,282Sep 3, 2024Updated 2 years ago
Are these results useful?