The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆126Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Proxied asynchronous multi-threaded web scraper via concurrent queues written in Java.☆17Nov 25, 2023Updated 2 years ago
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated last year
- Scrapyd on container infrastructure☆16Apr 11, 2025Updated last year
- HTTP proxy with per-request uTLS fingerprint mimicry and upstream proxy tunneling. Currently WIP.☆53Jan 14, 2024Updated 2 years ago
- A tool for pointing developers in the right direction for performance issues.☆12Apr 20, 2026Updated 2 weeks ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- (educational) build your own disk based KV store☆13Jul 27, 2024Updated last year
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆160Sep 28, 2025Updated 7 months ago
- ☆12Apr 16, 2025Updated last year
- An example backend with GoLang that uses auth0 for authentication☆18Jan 20, 2023Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- simhash php extension☆19Jan 6, 2017Updated 9 years ago
- ☆14Jun 19, 2024Updated last year
- This is a golang service template☆14Aug 10, 2025Updated 8 months ago
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Go SDK for working with Cerbos☆16Updated this week
- Sanity client for Go.☆18Apr 19, 2026Updated 2 weeks ago
- small MCP server for orchestrating tasks across LLM instances☆25Apr 29, 2025Updated last year
- 27.6% of the Top 10 Million Sites are Dead☆117Nov 4, 2024Updated last year
- An API to handle HTTP requests using a modified TLS Fingerprint.☆25Dec 21, 2024Updated last year
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆19Jul 11, 2025Updated 9 months ago
- 🚗 Real time package tracking implementation with RabbitMQ☆60Jul 13, 2022Updated 3 years ago
- "storycoin" -- distributed storytelling via proof-of-work blockchain☆10Feb 1, 2018Updated 8 years ago
- A decentralized poker game engine written in Golang and Solidity☆88Jan 27, 2024Updated 2 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Если сайт заблокирован, зеркало тут - https://storage.googleapis.com/amnezia/amnezia.org☆56Jul 11, 2025Updated 9 months ago
- ☆32Oct 30, 2025Updated 6 months ago
- Dockerized headless Chromium☆17Apr 29, 2026Updated last week
- Lightweight JavaScript library to interact with Chromium-based browsers via the Chrome DevTools Protocol☆27May 12, 2024Updated last year
- An obsolete python library which gathers statistics and relational information about Lean 3 libraries.☆17Mar 20, 2024Updated 2 years ago
- DIY home security project using Honeywell 5800 series RF sensors☆13Feb 12, 2020Updated 6 years ago
- A word game in the vein of Wordle; try to solve back-to-back code words to get to 100 points.☆27Feb 18, 2026Updated 2 months ago
- An advanced antibot for webdrivers☆278Dec 3, 2024Updated last year
- toy project to learn about the memory usage of different workloads with different allocators☆10Mar 30, 2023Updated 3 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Go version updater.☆22Jun 20, 2025Updated 10 months ago
- Small Helper Library to increase automatically the file descriptors limits for the current process☆24Jul 10, 2023Updated 2 years ago
- It contain google dork to find the wsdl file.☆13May 27, 2020Updated 5 years ago
- Search, sort, and filter Southwest flights based on a number of parameters.☆13Oct 29, 2024Updated last year
- 🌍🚀 Effortlessly simple i18n for Go. Plurals, gender, and more made easy!☆98Nov 19, 2023Updated 2 years ago
- A simple library that allows a network server to limit how may concurrent connections it supports from each client IP.☆54Apr 13, 2026Updated 3 weeks ago
- This application "listens" for a ticket creation event from Zendesk, analyses the ticket for negative sentiment, tags the ticket accordin…☆14Mar 10, 2025Updated last year