robotshell / robotScraper

RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
11Updated 4 months ago

Related projects

Alternatives and complementary repositories for robotScraper