The Architecture of a Web Crawler: Building a Google-Inspired Distributed Web Crawler
☆125Dec 11, 2024Updated last year
Alternatives and similar repositories for distributed-web-crawler
Users that are interested in distributed-web-crawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A puppeteer-extra plugin to solve Amazon captchas using Tessaract.JS.☆15May 16, 2024Updated 2 years ago
- tfidf provides TF-IDF functionality☆14Nov 4, 2023Updated 2 years ago
- Patching CDP (Chrome DevTools Protocol) leaks on OS level. Easy to use with Playwright, Selenium, and other web automation tools.☆164Sep 28, 2025Updated last year
- Key-value store on top of Raft Consensus Algorithm☆11Jun 12, 2018Updated 8 years ago
- A small library for building fast and highly customizable web crawlers☆16Jan 4, 2023Updated 3 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- DomainsProject.org HTTP worker☆25Dec 11, 2022Updated 3 years ago
- Library for creating genric data pipelines and streams☆11Dec 18, 2023Updated 2 years ago
- simhash php extension☆19Jan 6, 2017Updated 9 years ago
- This is a golang service template☆14Oct 5, 2026Updated last week
- DomainsProject.org DNS worker☆26Aug 11, 2024Updated 2 years ago
- A URL shortener written in Go, with a Mongo based backend, Prometheus and Grafana based monitoring, Memcached based write-through caching…☆38Jun 11, 2021Updated 5 years ago
- Hunter2 is a job hunt bot that indexes jobs and candidates from the fediverse☆14Jun 21, 2023Updated 3 years ago
- ☆17Nov 8, 2025Updated 11 months ago
- Efficient, idiomatic Go library for Backblaze B2 Cloud Storage.☆14Sep 13, 2018Updated 8 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- This repository is a mirror. See https://github.com/pantheon-systems/decoupled-kit-js to join the discussion or submit an issue.☆11Nov 14, 2025Updated 10 months ago
- Working draft to re-create USGS TNM Style Template for use in QGIS☆12Mar 21, 2019Updated 7 years ago
- Watsonx Assistant with Milvus as Vector Database☆11Mar 31, 2025Updated last year
- An API to handle HTTP requests using a modified TLS Fingerprint.☆25Dec 21, 2024Updated last year
- Python tools☆14Oct 22, 2023Updated 2 years ago
- Managing proxies for scaled data scraping and other automation operations will eventually require something like ProxyNest. ProxyNest is …☆22Feb 5, 2024Updated 2 years ago
- Dockerized headless Chromium☆16Aug 2, 2026Updated 2 months ago
- 🧩 Proposal to allow user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser en…☆20Jul 11, 2025Updated last year
- An advanced antibot for webdrivers☆287Dec 3, 2024Updated last year
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- ☆71Nov 17, 2023Updated 2 years ago
- Examples related to Amazon Lightsail☆12Jul 17, 2024Updated 2 years ago
- Feistel cipher implementation in Golang for format-preserving encryption☆18Jul 20, 2026Updated 2 months ago
- A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.☆102May 9, 2025Updated last year
- ☆32Oct 30, 2025Updated 11 months ago
- TikTok in Go☆13Mar 11, 2021Updated 5 years ago
- CI/CD repository template to automate deployments of your production flows☆15Jul 1, 2024Updated 2 years ago
- A single pipeline to deploy multiple services on amazon ecs using gitops☆41Sep 7, 2022Updated 4 years ago
- Chakra UI Animations is a dependancy which offers you pre-built animations for your Chakra UI components.☆14Oct 18, 2022Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Disaggregate zone-based origin/destination data to specific points☆13Oct 28, 2025Updated 11 months ago
- Example code demonstrating ecdh key agreement using openssl evp library.☆11Feb 20, 2018Updated 8 years ago
- 🚀 Web scraping for humans☆1,026Dec 1, 2024Updated last year
- This application "listens" for a ticket creation event from Zendesk, analyses the ticket for negative sentiment, tags the ticket accordin…☆14Mar 10, 2025Updated last year
- ☆11Apr 9, 2017Updated 9 years ago
- ☆13May 23, 2018Updated 8 years ago
- Implementation of Semantic Parsing with BERT and compositional pre-training on GeoQuery☆11Mar 20, 2019Updated 7 years ago