hueristiq / xcrawl3r

A command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpages, parsing resources like sitemaps and robots.txt files, and even processing local files - to uncover every URL.
96Updated last week

Alternatives and similar repositories for xcrawl3r:

Users that are interested in xcrawl3r are comparing it to the libraries listed below