ropensci / robotstxt

R 📦 for parsing and checking robots.txt files 🤖
68Updated 3 months ago

Alternatives and similar repositories for robotstxt:

Users that are interested in robotstxt are comparing it to the libraries listed below