The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆126Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- The Manta v1 software architecture for Autonomous Underwater Vehicles (AUVs) - Master's thesis☆10Aug 11, 2022Updated 3 years ago
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated 2 years ago
- A fullstack Rust + React chat app using open-source Llama language models☆34Sep 8, 2023Updated 2 years ago
- HTTP proxy with per-request uTLS fingerprint mimicry and upstream proxy tunneling. Currently WIP.☆54Jan 14, 2024Updated 2 years ago
- A simple JSON API that can fetch cat facts☆15Dec 14, 2022Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆161Sep 28, 2025Updated 8 months ago
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 7 years ago
- An example backend with GoLang that uses auth0 for authentication☆17Jan 20, 2023Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- simhash php extension☆19Jan 6, 2017Updated 9 years ago
- An extension of the UUV-Simulator for use with Vortex NTNUs autonomous vessels☆36May 17, 2023Updated 3 years ago
- This is a golang service template☆14Aug 10, 2025Updated 9 months ago
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- Graphon is a Python graph execution engine for agentic AI workflows.☆47Updated this week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Go SDK for working with Cerbos☆16Updated this week
- small MCP server for orchestrating tasks across LLM instances☆25Apr 29, 2025Updated last year
- A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching…☆38Jun 11, 2021Updated 4 years ago
- A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.☆101May 9, 2025Updated last year
- 27.6% of the Top 10 Million Sites are Dead☆117Nov 4, 2024Updated last year
- Run selenium undetected.☆32Mar 6, 2026Updated 2 months ago
- A beautiful, vibrant Neovim colorscheme inspired by spring blossoms with a soft, dreamy aesthetic.☆23Jan 5, 2026Updated 4 months ago
- Hunter2 is a job hunt bot that indexes jobs and candidates from the fediverse☆14Jun 21, 2023Updated 2 years ago
- ☆16Nov 8, 2025Updated 6 months ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- ORBIT - Interlink Remote Applications☆16May 19, 2026Updated last week
- "storycoin" -- distributed storytelling via proof-of-work blockchain☆10Feb 1, 2018Updated 8 years ago
- 🔮 Vindicate non-organic web traffic via MITM proxy☆90Jul 15, 2024Updated last year
- ☆14Feb 28, 2024Updated 2 years ago
- Working draft to re-create USGS TNM Style Template for use in QGIS☆11Mar 21, 2019Updated 7 years ago
- A decentralized poker game engine written in Golang and Solidity☆89Jan 27, 2024Updated 2 years ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆58May 20, 2026Updated last week
- FaaS (Function as a service) framework for writing portable R functions☆11Dec 31, 2020Updated 5 years ago
- A library for better integration between django and the WSGI world.☆50Jan 7, 2011Updated 15 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Dockerized headless Chromium☆17May 14, 2026Updated 2 weeks ago
- pyppeteer stealth plugin, attempts to look like a normal browser☆27Oct 3, 2024Updated last year
- ☆70Nov 17, 2023Updated 2 years ago
- A word game in the vein of Wordle; try to solve back-to-back code words to get to 100 points.☆27Feb 18, 2026Updated 3 months ago
- ☆20Jan 23, 2024Updated 2 years ago
- Infra-agnostic hosting framework for browser agents.☆154Sep 27, 2025Updated 8 months ago
- logging for django.☆35Jan 16, 2011Updated 15 years ago