temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
273Updated 2 years ago

Alternatives and similar repositories for robotstxt:

Users that are interested in robotstxt are comparing it to the libraries listed below