The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆125Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated 2 years ago
- Scrapyd on container infrastructure☆16May 29, 2026Updated 3 weeks ago
- HTTP proxy with per-request uTLS fingerprint mimicry and upstream proxy tunneling. Currently WIP.☆54Jan 14, 2024Updated 2 years ago
- A tool for pointing developers in the right direction for performance issues.☆12Apr 20, 2026Updated last month
- A simple JSON API that can fetch cat facts☆15Dec 14, 2022Updated 3 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 8 years ago
- An example backend with GoLang that uses auth0 for authentication☆17Jan 20, 2023Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- ☆14Jun 19, 2024Updated last year
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- Just a list of development tools that I find useful☆11Feb 17, 2018Updated 8 years ago
- Go SDK for working with Cerbos☆16Updated this week
- Get structured JSON data from any page.☆176Oct 11, 2023Updated 2 years ago
- A Collection of 10.000 collected Windows Chrome Fingerprints. Usable with an easy-to-use API, available as a compressed (lzma) or full-si…☆278Dec 22, 2024Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- small MCP server for orchestrating tasks across LLM instances☆25Apr 29, 2025Updated last year
- A proof of concept exploit that tries to pop you a shell on a uftpd server between versions 2.6 and 2.10☆20Apr 16, 2020Updated 6 years ago
- A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.☆101May 9, 2025Updated last year
- 27.6% of the Top 10 Million Sites are Dead☆117Nov 4, 2024Updated last year
- An API to handle HTTP requests using a modified TLS Fingerprint.☆25Dec 21, 2024Updated last year
- A prefix-enhanced map in Go☆31Jun 18, 2016Updated 10 years ago
- Run selenium undetected.☆32Mar 6, 2026Updated 3 months ago
- A CLI tool to get the latest news headlines from various news agencies☆11May 1, 2017Updated 9 years ago
- Efficient, idiomatic Go library for Backblaze B2 Cloud Storage.☆14Sep 13, 2018Updated 7 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆20Jul 11, 2025Updated 11 months ago
- ORBIT - Interlink Remote Applications☆16May 19, 2026Updated 3 weeks ago
- 🔮 Vindicate non-organic web traffic via MITM proxy☆91Jul 15, 2024Updated last year
- ☆12Mar 22, 2023Updated 3 years ago
- A decentralized poker game engine written in Golang and Solidity☆89Jan 27, 2024Updated 2 years ago
- ☆15Feb 13, 2024Updated 2 years ago
- Dockerized headless Chromium☆17May 24, 2026Updated 3 weeks ago
- pyppeteer stealth plugin, attempts to look like a normal browser☆27Oct 3, 2024Updated last year
- ☆70Nov 17, 2023Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- DIY home security project using Honeywell 5800 series RF sensors☆13Feb 12, 2020Updated 6 years ago
- An advanced antibot for webdrivers☆283Dec 3, 2024Updated last year
- ☆20Jan 23, 2024Updated 2 years ago
- Cash App CLI tool and TypeScript library☆11Sep 21, 2023Updated 2 years ago
- logging for django.☆35Jan 16, 2011Updated 15 years ago
- Terraform module to create Lightsail instance, Lightsail Key Pair (Optional), Lightsail Static IP (Optional), Metric alarm (Optional).☆22Updated this week
- A browser service with a REST API☆45Oct 20, 2024Updated last year