robotshell / robotScraper

RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
12Updated 5 months ago

Alternatives and similar repositories for robotScraper:

Users that are interested in robotScraper are comparing it to the libraries listed below