robotshell / robotScraperView on GitHub
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
15Jul 23, 2024Updated 2 years ago

Alternatives and similar repositories for robotScraper

Users that are interested in robotScraper are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

Sorting:

Are these results useful?