shavit / crawlero
Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
β16Updated 7 years ago
Alternatives and similar repositories for crawlero:
Users that are interested in crawlero are comparing it to the libraries listed below
- Google SEO scraper for "allintitle:keyword" queries.β23Updated 10 years ago
- Golang implementation of full-featured, lightweight and RFC compliant SMTP server.β10Updated 4 years ago
- πΈ A simple way to extract data from Common Crawlβ34Updated 5 years ago
- Analyzing social media sentiment and its impact on stock marketβ39Updated last year
- Example how to pre-process news articles with textbox and index on Elastic Searchβ13Updated 7 years ago
- Advanced declarative web scrapingβ30Updated 2 years ago
- Aiohttp web server API, which scrapes Google and returns scrape results as response. Supports proxies, multiple geos and number of resultβ¦β56Updated last year
- Site Hound (previously THH) is a Domain Discovery Toolβ23Updated 3 years ago
- build and send emailβ17Updated 2 years ago
- A distributed system for mining common crawl using SQS, AWS-EC2 and S3β18Updated 10 years ago
- Monitor your IP reputation for Email sending or Email marketing.β44Updated 11 years ago
- Automates the process of repeatedly searching for a website via scraped proxy IP and search keywordsβ44Updated last year
- Things I wish were Go built-insβ13Updated 4 years ago
- cloud-torrent and OpenVPN in a docker containerβ17Updated 7 years ago
- verify every email in fileβ16Updated 6 years ago
- Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trendsβ56Updated last year
- MySQL export to Elasticsearchβ14Updated 8 years ago
- WHOIS server builded in Golang and using Postgres as a DB.β11Updated 6 years ago
- Command Line Law - Contract management for developers | lawyersβ25Updated 6 years ago
- A browser extension that lets you find email addresses for any domain with a single click.β71Updated 7 years ago
- A Selenium based automated program that scrapes profiles data,stores in CSV,follows them and saves their profile in PDF.β32Updated last year
- Big Five personality traits: domains, aspects, facetsβ25Updated last year
- For SEO: checks a list of web pages if a backlink is presentβ11Updated 5 years ago
- Simple tool to import/export Elasticsearch indices into a file, and/or reshard an indexβ19Updated 3 years ago
- A list of 94,548 spam or temporary email domainβ12Updated 4 years ago
- Self-destructing notes on Go with tiny secured client-sideβ29Updated 2 years ago
- β14Updated 2 years ago
- Get results from search engines.β12Updated 2 years ago
- Phantombuster's SDKβ14Updated 5 months ago
- Crawler and scraper of the public directory of companies on LinkedIn.β25Updated 5 years ago