scrapy / protegoLinks

A pure-Python robots.txt parser with support for modern conventions.
65Updated 2 months ago

Alternatives and similar repositories for protego

Users that are interested in protego are comparing it to the libraries listed below

Sorting: