scrapy / protegoLinks

A pure-Python robots.txt parser with support for modern conventions.
70Updated last week

Alternatives and similar repositories for protego

Users that are interested in protego are comparing it to the libraries listed below

Sorting: