The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆126Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A simple JSON API that can fetch cat facts☆15Dec 14, 2022Updated 3 years ago
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated 2 years ago
- Real-Time IP Location Visualization for Network TCP Packets☆13Jul 3, 2024Updated 2 years ago
- Organise Go http routes using structs☆24Updated this week
- This repository showcases a universal web scraping agent powered by Large Language Models (LLMs), simplifying web scraping by abstracting…☆11Jun 3, 2024Updated 2 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 8 years ago
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆163Sep 28, 2025Updated last year
- An example backend with GoLang that uses auth0 for authentication☆17Jan 20, 2023Updated 3 years ago
- A small library for building fast and highly customizable web crawlers☆16Jan 4, 2023Updated 3 years ago
- A B+ Tree based database storage engine, written in Go.☆13Sep 13, 2026Updated last month
- This is a golang service template☆14Oct 24, 2026Updated last week
- DomainsProject.org HTTP worker☆25Dec 11, 2022Updated 3 years ago
- API for https://github.com/03balogun/docai-web☆10Jul 18, 2023Updated 3 years ago
- An extension of the UUV-Simulator for use with Vortex NTNUs autonomous vessels☆39May 17, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- A CLI tool for storing and searching development memories, insights, and patterns using Cursor Rules, Supabase, and OpenAI embeddings.☆15Aug 4, 2025Updated last year
- MetaDataScraper is a Python package designed to automate the extraction of follower counts and post details from a public Facebook page. …☆13Aug 23, 2024Updated 2 years ago
- Examples using encoding/binary package☆16Dec 12, 2021Updated 4 years ago
- A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching…☆38Jun 11, 2021Updated 5 years ago
- Efficient, idiomatic Go library for Backblaze B2 Cloud Storage.☆14Sep 13, 2018Updated 8 years ago
- Sanity client for Go.☆20Oct 24, 2026Updated last week
- Proof of History in Golang. Taking key concepts from the Solana whitepaper and providing examples in Go☆14Jan 9, 2022Updated 4 years ago
- A Python library for routing and handling websocket messages using django-channels.☆11Jul 17, 2017Updated 9 years ago
- 😈📚 A curated library of research papers and presentations for counter-detection and web privacy enthusiasts.☆774Feb 19, 2024Updated 2 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Patch binary file using IDA signatures and defined replacement bytes in YAML.☆10Feb 28, 2025Updated last year
- A decentralized poker game engine written in Golang and Solidity☆89Jan 27, 2024Updated 2 years ago
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆20Jul 11, 2025Updated last year
- Python tools☆14Oct 22, 2023Updated 3 years ago
- Open source desktop app and Chromium extension for capturing, organizing, and verifying web evidence in OSINT investigations. Local-first…☆17Updated this week
- ☆20Jan 23, 2024Updated 2 years ago
- Search, sort, and filter Southwest flights based on a number of parameters.☆13Oct 29, 2024Updated 2 years ago
- Go version updater.☆22Jun 20, 2025Updated last year
- Links or additional data to some researches☆13May 24, 2019Updated 7 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- 🦀 Telegram Bot to notify when a website is down. Health check.☆17Oct 12, 2026Updated 3 weeks ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆60Oct 26, 2026Updated last week
- An API to handle HTTP requests using a modified TLS Fingerprint.☆25Dec 21, 2024Updated last year
- 🔮 Vindicate non-organic web traffic via MITM proxy☆92Jul 15, 2024Updated 2 years ago
- Lightweight JavaScript library to interact with Chromium-based browsers via the Chrome DevTools Protocol☆29Sep 23, 2026Updated last month
- PHP Accelerated File Download With Axel☆10Jun 6, 2015Updated 11 years ago
- typing speed test tui☆15Mar 18, 2026Updated 7 months ago