The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆125Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- tfidf provides TF-IDF functionality☆14Nov 4, 2023Updated 2 years ago
- HTTP proxy with per-request uTLS fingerprint mimicry and upstream proxy tunneling. Currently WIP.☆54Jan 14, 2024Updated 2 years ago
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆162Sep 28, 2025Updated 9 months ago
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 8 years ago
- An example backend with GoLang that uses auth0 for authentication☆17Jan 20, 2023Updated 3 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated last year
- Sanity client for Go.☆19Jun 7, 2026Updated last month
- Pyvigate: A Python framework that combines headless browsing with LLMs that assists you in your data solutions, product tours, building R…☆27Mar 6, 2024Updated 2 years ago
- A Collection of 10.000 collected Windows Chrome Fingerprints. Usable with an easy-to-use API, available as a compressed (lzma) or full-si…☆280Dec 22, 2024Updated last year
- small MCP server for orchestrating tasks across LLM instances☆25Apr 29, 2025Updated last year
- A proof of concept exploit that tries to pop you a shell on a uftpd server between versions 2.6 and 2.10☆20Apr 16, 2020Updated 6 years ago
- A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching…☆38Jun 11, 2021Updated 5 years ago
- 27.6% of the Top 10 Million Sites are Dead☆117Nov 4, 2024Updated last year
- Run selenium undetected.☆32Mar 6, 2026Updated 4 months ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Hunter2 is a job hunt bot that indexes jobs and candidates from the fediverse☆14Jun 21, 2023Updated 3 years ago
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆20Jul 11, 2025Updated 11 months ago
- 🚗 Real time package tracking implementation with RabbitMQ☆60Jul 13, 2022Updated 3 years ago
- ORBIT - Interlink Remote Applications☆16May 19, 2026Updated last month
- Working draft to re-create USGS TNM Style Template for use in QGIS☆12Mar 21, 2019Updated 7 years ago
- Python tools☆14Oct 22, 2023Updated 2 years ago
- Airbnb scraper made in Go☆39Jul 10, 2025Updated 11 months ago
- PoC to see if integration testing with Jest is doable with Step Functions Local☆14Dec 15, 2023Updated 2 years ago
- Repository for the Mastering Web Scraping in Python: Scaling to Distributed Crawling blogpost with the final code.☆46Oct 29, 2021Updated 4 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Lightweight JavaScript library to interact with Chromium-based browsers via the Chrome DevTools Protocol☆27May 12, 2024Updated 2 years ago
- pyppeteer stealth plugin, attempts to look like a normal browser☆27Oct 3, 2024Updated last year
- ☆70Nov 17, 2023Updated 2 years ago
- DIY home security project using Honeywell 5800 series RF sensors☆13Feb 12, 2020Updated 6 years ago
- An advanced antibot for webdrivers☆283Dec 3, 2024Updated last year
- TikTok in Go☆13Mar 11, 2021Updated 5 years ago
- Terraform module to create Lightsail instance, Lightsail Key Pair (Optional), Lightsail Static IP (Optional), Metric alarm (Optional).☆22Jun 15, 2026Updated 3 weeks ago
- A single pipeline to deploy multiple services on amazon ecs using gitops☆41Sep 7, 2022Updated 3 years ago
- Search the web with advanced filters and LLM-friendly output formats!☆58Sep 15, 2025Updated 9 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Example code demonstrating ecdh key agreement using openssl evp library.☆11Feb 20, 2018Updated 8 years ago
- vader sentiment analysis in go☆55Apr 29, 2025Updated last year
- Zippy Talking Avatar uses Azure Cognitive Services and OpenAI API to generate text and speech. It is built with Next.js and Tailwind CSS.…☆16Feb 9, 2024Updated 2 years ago
- This application "listens" for a ticket creation event from Zendesk, analyses the ticket for negative sentiment, tags the ticket accordin…☆14Mar 10, 2025Updated last year
- teler Caddy integrates the powerful security features of teler WAF into the Caddy web server, ensuring your web servers remain secure and…☆17Feb 24, 2025Updated last year
- declarative flag parsing for Go using struct tags☆12May 7, 2023Updated 3 years ago
- ☆35Nov 25, 2023Updated 2 years ago