scrapy / protegoLinks

A pure-Python robots.txt parser with support for modern conventions.
69Updated this week

Alternatives and similar repositories for protego

Users that are interested in protego are comparing it to the libraries listed below

Sorting: