aichat-bot / crawlyLinks

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
25Updated last year

Alternatives and similar repositories for crawly

Users that are interested in crawly are comparing it to the libraries listed below

Sorting: