shavit / crawlero
Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
☆16Updated 7 years ago
Alternatives and similar repositories for crawlero:
Users that are interested in crawlero are comparing it to the libraries listed below
- A set of services for monitoring of multiple social media platforms based on Docker.☆16Updated 3 years ago
- Analyzing social media sentiment and its impact on stock market☆39Updated last year
- Social media monitoring tools such as sentiment analysis, keyword tracking and more☆47Updated 11 years ago
- Monitor your IP reputation for Email sending or Email marketing.☆43Updated 11 years ago
- Simple application written in Go that combines two wordlists and a list of TLDs to form domain names and check if they are already regist…☆27Updated 7 years ago
- WHOIS server builded in Golang and using Postgres as a DB.☆11Updated 6 years ago
- Simple tool to import/export Elasticsearch indices into a file, and/or reshard an index☆18Updated 2 years ago
- Site Hound (previously THH) is a Domain Discovery Tool☆23Updated 3 years ago
- Catch the dropped domains with public api from Godaddy,Name,NameSilo,Dynadot etc☆50Updated 7 years ago
- Sherry is an asynchronous and distributed task queue in Go☆13Updated 5 years ago
- A Selenium based automated program that scrapes profiles data,stores in CSV,follows them and saves their profile in PDF.☆32Updated last year
- Get results from search engines.☆12Updated 2 years ago
- The basics of Go with the Colly framework☆8Updated 5 years ago
- Check the DNS records for a domain and check worldwide propagation of changes that you made.☆17Updated 5 years ago
- Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords☆43Updated last year
- A simple tool to scan your website to keep your cache hot & ready. Helper tool for Prerender, Squid, CDN etc..☆13Updated 6 years ago
- build and send email☆17Updated last year
- Advanced declarative web scraping☆30Updated last year
- Extensions for using Scrapy on Amazon AWS☆32Updated 12 years ago
- Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends☆56Updated 11 months ago
- keywords-extract - Command line tool extract keywords from any web page.☆63Updated 6 years ago
- Simple program that summarize text.☆10Updated 14 years ago
- A browser extension that lets you find email addresses for any domain with a single click.☆71Updated 7 years ago
- Google SEO scraper for "allintitle:keyword" queries.☆23Updated 10 years ago
- Monitor domain names for DNS changes☆9Updated last year
- Object storage microservice. Like minio but minnier.☆9Updated 5 years ago