stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
226Updated last year

Related projects

Alternatives and complementary repositories for cobweb