google / robotstxtLinks

The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
3,410Updated 10 months ago

Alternatives and similar repositories for robotstxt

Users that are interested in robotstxt are comparing it to the libraries listed below

Sorting: