FCC / Crawler

Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.
90Updated 12 years ago

Alternatives and similar repositories for Crawler:

Users that are interested in Crawler are comparing it to the libraries listed below