Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.
☆438Dec 30, 2022Updated 3 years ago
Alternatives and similar repositories for Crawling-Infrastructure
Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Cloud crawler functions for scrapeulous☆44Feb 24, 2021Updated 5 years ago
- Javascript scraping module based on puppeteer for many different search engines...☆570Dec 30, 2022Updated 3 years ago
- Minimal set of tools to conduct stealthy scraping.☆166Apr 21, 2023Updated 3 years ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆70Jun 8, 2021Updated 5 years ago
- 🧱 A uniform template to use as a foundation for Puppeteer bot construction.☆68May 6, 2021Updated 5 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Scrapoxy has been discontinued.☆2,409Feb 7, 2026Updated 8 months ago
- In-Memory Key-Value Database with Persistent File Storage☆16Sep 24, 2022Updated 4 years ago
- 💯 Teach puppeteer new tricks through plugins.☆7,406Jul 18, 2024Updated 2 years ago
- Passive TCP/IP Fingerprinting Tool. Run this on your server and find out what Operating Systems your clients are *really* using.☆447Sep 21, 2026Updated 2 weeks ago
- Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprint…☆5,150Jul 27, 2026Updated 2 months ago
- Solution to stop sites from fingerprinting your puppeteer☆131Jun 12, 2026Updated 3 months ago
- ☆174Dec 30, 2022Updated 3 years ago
- The web scraper that's nearly impossible to block - now called @ulixee/hero☆733Mar 7, 2023Updated 3 years ago
- POC code to crash Windows Event Logger Service☆27Oct 16, 2020Updated 5 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Repo for hosting various scripts for creating users for password spraying and other password attacks.☆11Jul 9, 2020Updated 6 years ago
- #️⃣ 🕸️ 👤 HTTP Headers Hashing☆12Aug 27, 2023Updated 3 years ago
- Distributed crawler powered by Headless Chrome☆5,632Apr 29, 2023Updated 3 years ago
- Microsoft Applocker evasion tool☆39Nov 26, 2019Updated 6 years ago
- Docker kinsing malware bitcoin/xmr miner☆20Feb 18, 2021Updated 5 years ago
- Event Data Collector☆40Updated this week
- A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…☆382Dec 30, 2022Updated 3 years ago
- BH Cypher Queries picked up from random places☆41Dec 12, 2018Updated 7 years ago
- List of free and checked http, https, socks4 and socks5 proxies☆23Updated this week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data …☆26,073Updated this week
- Article extraction benchmark: dataset and evaluation scripts☆378May 29, 2026Updated 4 months ago
- A test suite of common scraper detection techniques. See how detectable your scraper stack is.☆138Oct 31, 2022Updated 3 years ago
- 👀 Analyze Websites and Resources They Request☆24Feb 3, 2019Updated 7 years ago
- How to detect puppeteer with 100% accuracy☆107May 30, 2021Updated 5 years ago
- interactive command line interfaces for Python☆13Jan 3, 2021Updated 5 years ago
- ☆53Oct 20, 2020Updated 5 years ago
- Scrapy rotation proxy package with advanced functions☆94Jul 4, 2022Updated 4 years ago
- LazyText is inspired by the idea of lazypredict, a library which helps build lot of basic models without much code. LazyText is for text …☆18Feb 19, 2022Updated 4 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Automated WireGuard Deployment on Azure☆46Feb 28, 2021Updated 5 years ago
- my static site blog using pelican☆22Aug 29, 2026Updated last month
- documentation on the data-dome captcha payload☆29Feb 1, 2023Updated 3 years ago
- DFPM is a browser extension for detecting browser fingerprinting.☆126Dec 6, 2022Updated 3 years ago
- Lightweight, scriptable browser as a service with an HTTP API☆4,191Aug 2, 2024Updated 2 years ago
- Collection of IronPython scripts and executables for penetration testing☆57Feb 26, 2019Updated 7 years ago
- A beautiful way to share code☆14Mar 4, 2023Updated 3 years ago