aichat-bot / crawly
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
☆24Updated 10 months ago
Alternatives and similar repositories for crawly:
Users that are interested in crawly are comparing it to the libraries listed below
- Server-sent events (SSE) client implementation for Rust☆85Updated last month
- User Agent Parser for Rust☆45Updated 8 months ago
- Simple ACME client☆122Updated last week
- Helper to build an Eventsource using reqwest☆57Updated 10 months ago
- A Rust crate for extracting readable content from web pages.☆171Updated last week
- A Rust crate providing an implementation of an RFC-compliant `EmailAddress` newtype.☆72Updated 5 months ago
- JWT authorization layer for Axum.☆76Updated 8 months ago
- Simple string matching with single- and multiple-wildcard operator☆84Updated 7 months ago
- Common stop words in a variety of languages☆21Updated 2 months ago
- A token-based rate limiter based on the leaky bucket algorithm.☆108Updated last month
- More efficient alternative to `serde_json::Value` which saves memory by interning primitive values and using tagged pointers.☆135Updated 5 months ago
- HTML to markdown for Rust☆35Updated 2 months ago
- This crate provides functions for working with IPv4 CIDRs and IPv6 CIDRs.☆32Updated last year
- DKIM, ARC, SPF and DMARC library for Rust☆98Updated last month
- Various experiments written in order to learn more about how Tremor's plugin system could be implemented☆29Updated 2 years ago
- Graceful shutdown util for Rust projects using the Tokio Async runtime.☆135Updated 7 months ago
- Distributed locks with Redis, implemented in Rust☆54Updated 7 months ago
- Opinionated framework for web services☆30Updated last week
- Reports when tokio runtime threads are blocking☆64Updated 8 months ago
- High-level http auth extractors for axum☆111Updated 3 months ago
- Intercepting MitM proxy with flexible security scanner power☆53Updated 6 months ago
- A very fast HTTP Server based on Hyper and Rust☆86Updated last year
- Background jobs inspector for Apalis☆26Updated 5 months ago
- A middleware for actix-web that provides rate-limiting backed by governor.☆110Updated 3 months ago
- waPC Rust monorepo☆81Updated this week
- Streaming JSON reader and writer written in Rust☆72Updated last week
- Task runner and process manager for Rust☆50Updated 7 months ago
- Structures and tools to parse, navigate and validate OpenAPI v3.1 specifications.☆54Updated this week
- A problem type to be returned by HTTP APIs☆59Updated 4 months ago
- ☆40Updated last month