The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆126Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated last year
- (educational) build your own disk based KV store☆13Jul 27, 2024Updated last year
- A simple JSON API that can fetch cat facts☆15Dec 14, 2022Updated 3 years ago
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆160Sep 28, 2025Updated 6 months ago
- ☆12Apr 16, 2025Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- A small library for building fast and highly customizable web crawlers☆16Jan 4, 2023Updated 3 years ago
- An example backend with GoLang that uses auth0 for authentication☆18Jan 20, 2023Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- https://mids-w203.github.io/practice_problems/☆12Feb 25, 2026Updated last month
- A CSP (Communicating Sequential Processes) written in TypeScript, based on Paybase's csp library☆10Mar 6, 2023Updated 3 years ago
- Go SDK for working with Cerbos☆16Updated this week
- Get structured JSON data from any page.☆176Oct 11, 2023Updated 2 years ago
- Sanity client for Go.☆18Updated this week
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- small MCP server for orchestrating tasks across LLM instances☆24Apr 29, 2025Updated 11 months ago
- A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching…☆39Jun 11, 2021Updated 4 years ago
- A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.☆98May 9, 2025Updated 11 months ago
- 27.6% of the Top 10 Million Sites are Dead☆117Nov 4, 2024Updated last year
- An API to handle HTTP requests using a modified TLS Fingerprint.☆25Dec 21, 2024Updated last year
- 🚗 Real time package tracking implementation with RabbitMQ☆60Jul 13, 2022Updated 3 years ago
- ORBIT - Interlink Remote Applications☆16Jan 12, 2026Updated 3 months ago
- A decentralized poker game engine written in Golang and Solidity☆88Jan 27, 2024Updated 2 years ago
- ☆17Dec 16, 2020Updated 5 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A library for better integration between django and the WSGI world.☆50Jan 7, 2011Updated 15 years ago
- A word game in the vein of Wordle; try to solve back-to-back code words to get to 100 points.☆27Feb 18, 2026Updated 2 months ago
- ☆24Mar 16, 2019Updated 7 years ago
- Go version updater.☆22Jun 20, 2025Updated 9 months ago
- cf_clearance, rack.session, laravel_session, _bm etc... cloudflare cookies generator via headless browser☆32Feb 4, 2023Updated 3 years ago
- It contain google dork to find the wsdl file.☆13May 27, 2020Updated 5 years ago
- Fast, zero-configuration, static HTTP filer server.☆11Apr 16, 2025Updated last year
- vader sentiment analysis in go☆54Apr 29, 2025Updated 11 months ago
- grobotstxt is a native Go port of Google's robots.txt parser and matcher library.☆115Mar 16, 2022Updated 4 years ago
- Serverless GPU API endpoints on Runpod - Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- This application "listens" for a ticket creation event from Zendesk, analyses the ticket for negative sentiment, tags the ticket accordin…☆14Mar 10, 2025Updated last year
- 🚀 Web scraping for humans☆1,001Dec 1, 2024Updated last year
- A complete open source e-commerce solution built with Rust(STILL IN DEVELOPMENT).☆11Jul 29, 2018Updated 7 years ago
- teler Caddy integrates the powerful security features of teler WAF into the Caddy web server, ensuring your web servers remain secure and…☆17Feb 24, 2025Updated last year
- declarative flag parsing for Go using struct tags☆12May 7, 2023Updated 2 years ago
- A lightweight, dependency-free setup for git & ZShell.☆20May 28, 2025Updated 10 months ago
- ☆22Aug 1, 2025Updated 8 months ago