aichat-bot / crawly

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
21Updated 7 months ago

Alternatives and similar repositories for crawly:

Users that are interested in crawly are comparing it to the libraries listed below