p0dalirius / robotstester

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
42Updated last month

Alternatives and similar repositories for robotstester:

Users that are interested in robotstester are comparing it to the libraries listed below