Alternatives and similar repositories for distributed-web-crawler:

Users that are interested in distributed-web-crawler are comparing it to the libraries listed below

• spider-rs / spider-py
  Spider ported to Python
  ☆63Updated 3 months ago
• gosom / scrapemate
  Golang Crawling and scraping framework
  ☆97Updated this week
• getdera / dera
  Data Encoding and Representation Analysis
  ☆40Updated 11 months ago
• tonywangcn / ten-million-domains
  27.6% of the Top 10 Million Sites are Dead
  ☆101Updated 2 months ago
• johnbalvin / gobnb
  Airbnb scraper made in Go
  ☆34Updated 8 months ago
• semanser / JsonGenius
  Get structured JSON data from any page.
  ☆175Updated last year
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆73Updated 7 months ago
• DataDog / documentor
  Improve technical documentation with the power of AI.
  ☆22Updated 2 weeks ago
• cullenwatson / StaffSpy
  Staff scraper library for LinkedIn - obtain experiences, schools, skills & contact info
  ☆90Updated 2 weeks ago
• TheWebScrapingClub / TheScrapingClubFree
  The Web Scraping Club Free Repository
  ☆135Updated 2 months ago
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆40Updated last month
• missingstudio / gateway
  [deprecated] AI Gateway - core infrastructure stack for building production-ready AI Applications
  ☆155Updated 9 months ago
• russellballestrini / opencompletion
  Chatroom app where messages are sent to GPT, Claude, Mistral, Together, Grok, Groq, vLLM, Ollama & streamed to the frontend.
  ☆38Updated 2 weeks ago
• PxyUp / fitter
  New way for collect information from the API's/Websites
  ☆120Updated last month
• sedpeep / Twitter-Scraper
  Extract tweets based on the search query and extracts the results and store in selenium
  ☆41Updated 5 months ago
• ruvnet / guardrail
  GuardRail: Advanced tool for data analysis and AI content generation using OpenAI GPT models. Features sentiment analysis, content classi…
  ☆125Updated last year
• tech-engine / goscrapy
  GoScrapy: Harnessing Go's power for blazingly fast web scraping, inspired by Python's Scrapy framework.
  ☆91Updated last month
• yisding / libchatty
  ☆21Updated 3 months ago
• struct-chat / embedding
  Vector Embedding Server in under 100 lines of code
  ☆22Updated 10 months ago
• ukanwat / ScriptGPT
  ScriptGPT turns your ideas into JS/TS functional code with the power of GPT4
  ☆22Updated last year
• devinterface / startersaas-go-api
  Golang API for a SaaS boilerplate
  ☆55Updated last year
• customeros / mailsherpa
  CLI to verify an if an email address is deliverable. Uses SMTP to validate email addresses without sending an email.
  ☆18Updated last month
• whitew1994WW / AutoPA
  ☆12Updated 5 months ago
• LyzrCore / lyzr
  Lyzr SDKs help you to build all your favorite GenAI SaaS products as enterprise applications in minutes.
  ☆169Updated last month
• ryszard / agency
  Agency: Robust LLM Agent Management with Go
  ☆61Updated 9 months ago
• couldbejake / fast
  Proxied asynchronous multi-threaded web scraper via concurrent queues written in Java.
  ☆16Updated last year
• pluja / maestro
  Turn natual language into commands. Your CLI tasks, now as easy as a conversation. Run it 100% offline, or use OpenAI's models.
  ☆55Updated 6 months ago
• msoedov / vector_lake
  S3 vector database for LLM Agents and RAG.
  ☆35Updated last year
• yoheinakajima / jsondr
  converts url content into JSON with a simple prefix
  ☆64Updated 8 months ago
• bitmakerla / estela
  estela, an elastic web scraping cluster 🕸
  ☆175Updated 2 months ago