robotshell / robotScraper

RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
13Updated 6 months ago

Alternatives and similar repositories for robotScraper:

Users that are interested in robotScraper are comparing it to the libraries listed below