The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆125Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆16May 16, 2024Updated 2 years ago
- Scrapyd on container infrastructure☆16May 29, 2026Updated 2 months ago
- A simple JSON API that can fetch cat facts☆15Dec 14, 2022Updated 3 years ago
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆164Sep 28, 2025Updated 10 months ago
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 8 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A small library for building fast and highly customizable web crawlers☆16Jan 4, 2023Updated 3 years ago
- An example backend with GoLang that uses auth0 for authentication☆17Jan 20, 2023Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- A CLI tool for storing and searching development memories, insights, and patterns using Cursor Rules, Supabase, and OpenAI embeddings.☆15Aug 4, 2025Updated 11 months ago
- A Collection of 10.000 collected Windows Chrome Fingerprints. Usable with an easy-to-use API, available as a compressed (lzma) or full-si…☆287Dec 22, 2024Updated last year
- Efficient, idiomatic Go library for Backblaze B2 Cloud Storage.☆14Sep 13, 2018Updated 7 years ago
- Python tools☆14Oct 22, 2023Updated 2 years ago
- "storycoin" -- distributed storytelling via proof-of-work blockchain☆10Feb 1, 2018Updated 8 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- FaaS (Function as a service) framework for writing portable R functions☆12Dec 31, 2020Updated 5 years ago
- Managing proxies for scaled data scraping and other automation operations will eventually require something like ProxyNest. ProxyNest is …☆22Feb 5, 2024Updated 2 years ago
- Run selenium undetected.☆32Mar 6, 2026Updated 4 months ago
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆20Jul 11, 2025Updated last year
- An advanced antibot for webdrivers☆290Dec 3, 2024Updated last year
- ☆19Jan 23, 2024Updated 2 years ago
- A word game in the vein of Wordle; try to solve back-to-back code words to get to 100 points.☆28Jul 17, 2026Updated last week
- Docker container to mount rclone remotes over FUSE☆12Aug 24, 2021Updated 4 years ago
- pyppeteer stealth plugin, attempts to look like a normal browser☆27Oct 3, 2024Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Zippy Talking Avatar uses Azure Cognitive Services and OpenAI API to generate text and speech. It is built with Next.js and Tailwind CSS.…☆16Feb 9, 2024Updated 2 years ago
- Disaggregate zone-based origin/destination data to specific points☆13Oct 28, 2025Updated 9 months ago
- cf_clearance, rack.session, laravel_session, _bm etc... cloudflare cookies generator via headless browser☆32Feb 4, 2023Updated 3 years ago
- Fast subdomain bruteforce enumerator in PHP 7.1☆34May 4, 2017Updated 9 years ago
- 🔮 Vindicate non-organic web traffic via MITM proxy☆92Jul 15, 2024Updated 2 years ago
- Framework which makes large scale crawling of URLs with VisibleV8 easy.☆13Jul 20, 2026Updated last week
- Hydrology and Climate Forecasting R package☆16Mar 9, 2025Updated last year
- A proof-of-concept of hybrid automation running the browser!☆17Jun 19, 2024Updated 2 years ago
- ☆35Nov 25, 2023Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- This application "listens" for a ticket creation event from Zendesk, analyses the ticket for negative sentiment, tags the ticket accordin…☆14Mar 10, 2025Updated last year
- A Distributed Synchronization Library for Golang Built On Top of Object Storage.☆14Apr 18, 2024Updated 2 years ago
- ☆18Feb 1, 2025Updated last year
- Modern GIS Web Client for JavaScript, based on MapboxGL-JS, OpenLayers, Leaflet☆13Sep 16, 2022Updated 3 years ago
- A self-hosted journal and article archiver with a gallery feature built on top of Django, that enables collaboration and note-taking.☆11Jul 7, 2026Updated 3 weeks ago
- Tools to create a map of urban accessibility using OpenStreetMap data and QGIS☆13Jun 6, 2025Updated last year
- ☆11Aug 16, 2021Updated 4 years ago