Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.
☆438Dec 30, 2022Updated 3 years ago
Alternatives and similar repositories for Crawling-Infrastructure
Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Cloud crawler functions for scrapeulous☆44Feb 24, 2021Updated 5 years ago
- Javascript scraping module based on puppeteer for many different search engines...☆570Dec 30, 2022Updated 3 years ago
- Minimal set of tools to conduct stealthy scraping.☆166Apr 21, 2023Updated 3 years ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆70Jun 8, 2021Updated 5 years ago
- A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.☆2,878Jul 3, 2021Updated 5 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- ☆12May 7, 2023Updated 3 years ago
- 🧱 A uniform template to use as a foundation for Puppeteer bot construction.☆68May 6, 2021Updated 5 years ago
- This repository contains instructions how to use the free IP Address API. The databases are: ASN database, Geolocation database, hosting …☆119Aug 1, 2026Updated last month
- Scrapoxy has been discontinued.☆2,414Feb 7, 2026Updated 6 months ago
- In-Memory Key-Value Database with Persistent File Storage☆16Sep 24, 2022Updated 3 years ago
- 💯 Teach puppeteer new tricks through plugins.☆7,398Jul 18, 2024Updated 2 years ago
- Search google, bing, yahoo, and other search engines with python☆669Apr 2, 2025Updated last year
- Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprint…☆5,131Jul 27, 2026Updated last month
- Solution to stop sites from fingerprinting your puppeteer☆130Jun 12, 2026Updated 2 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- The web scraper that's nearly impossible to block - now called @ulixee/hero☆738Mar 7, 2023Updated 3 years ago
- ☆20Apr 21, 2020Updated 6 years ago
- Repo for hosting various scripts for creating users for password spraying and other password attacks.☆11Jul 9, 2020Updated 6 years ago
- Microsoft Applocker evasion tool☆39Nov 26, 2019Updated 6 years ago
- Docker kinsing malware bitcoin/xmr miner☆20Feb 18, 2021Updated 5 years ago
- Event Data Collector☆40Mar 23, 2026Updated 5 months ago
- Puppeteer Pool, run a cluster of instances in parallel☆3,513Mar 1, 2026Updated 6 months ago
- Site Hound (previously THH) is a Domain Discovery Tool☆24Apr 8, 2026Updated 4 months ago
- ☆10Dec 18, 2018Updated 7 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- List of free and checked http, https, socks4 and socks5 proxies☆23Updated this week
- Nodejs lib to parse Google SERP html pages☆48Jul 27, 2023Updated 3 years ago
- A JavaScript library for generating random user agents with data that's updated daily.☆1,188Updated this week
- ☆117Mar 16, 2024Updated 2 years ago
- Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data …☆25,634Updated this week
- A simple, quick, and dirty websocket shell for PowerShell.☆20Jun 5, 2017Updated 9 years ago
- A test suite of common scraper detection techniques. See how detectable your scraper stack is.☆139Oct 31, 2022Updated 3 years ago
- BloodHound Cypher Queries Ported to a Jupyter Notebook☆53Jun 20, 2020Updated 6 years ago
- 👀 Analyze Websites and Resources They Request☆24Feb 3, 2019Updated 7 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆53Oct 20, 2020Updated 5 years ago
- This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.☆1,226Nov 7, 2023Updated 2 years ago
- OSINT Resources for Politics☆14Aug 13, 2018Updated 8 years ago
- Scrapy rotation proxy package with advanced functions☆94Jul 4, 2022Updated 4 years ago
- Additional module to use with 'puppeteer' for setting proxies per page basis.☆449Jun 9, 2024Updated 2 years ago
- my static site blog using pelican☆22Updated this week
- documentation on the data-dome captcha payload☆29Feb 1, 2023Updated 3 years ago