stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
226Updated 2 years ago

Alternatives and similar repositories for cobweb:

Users that are interested in cobweb are comparing it to the libraries listed below