shavit / crawlero

Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
16Updated 7 years ago

Alternatives and similar repositories for crawlero:

Users that are interested in crawlero are comparing it to the libraries listed below