scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
60Updated this week

Alternatives and similar repositories for protego:

Users that are interested in protego are comparing it to the libraries listed below