aichat-bot / crawlyLinks
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
☆25Updated last year
Alternatives and similar repositories for crawly
Users that are interested in crawly are comparing it to the libraries listed below
Sorting:
- Simple string matching with single- and multiple-wildcard operator☆88Updated last month
- Streaming JSON reader and writer written in Rust☆83Updated last week
- A Rust crate for extracting readable content from web pages.☆180Updated last week
- A very fast Key-Value store in pure rust☆210Updated 3 months ago
- Fake data generator library with 130+ functions written in Rust☆171Updated 9 months ago
- Goku is an HTTP load testing application written in Rust☆139Updated 4 months ago
- Simple, minimal-dependency HTTP client.☆238Updated last month
- A middleware for actix-web that provides rate-limiting backed by governor.☆120Updated last week
- IP2Location DB Reader☆32Updated last month
- An extremely dev-friendly, composable messaging platform☆57Updated last year
- A token-based rate limiter based on the leaky bucket algorithm.☆112Updated last week
- A very fast HTTP Server based on Hyper and Rust☆87Updated last year
- Rust PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.☆65Updated 10 months ago
- Structures and tools to parse, navigate and validate OpenAPI v3.1 specifications.☆63Updated last week
- ⏱️ An asynchronous task scheduling library written in Rust☆74Updated this week
- Generate client libraries that are featureful, human, well-documented, and async based on OpenAPI specs☆92Updated 9 months ago
- A tracing layer for shipping logs to Grafana Loki☆67Updated 4 months ago
- Rust web servers without async/await.☆217Updated 9 months ago
- Parallelo Parallel Library (PPL) is a small parallel framework that brings Structured Parallel Programming in Rust.☆78Updated last month
- Simple ACME client☆163Updated this week
- Wildcard matching☆202Updated 2 weeks ago
- A collection of different transport implementations and related tools focused primarily on web technologies.☆321Updated this week
- Command line interface for the Phylum API☆102Updated this week
- Intercepting MitM proxy with flexible security scanner power☆54Updated last year
- Rust bindings for Tesseract☆161Updated 6 months ago
- BUS/RT - Modern, fast, Rust-native IPC broker☆125Updated 3 months ago
- A crate to separate your HTML from your Rust code when using Yew☆42Updated 2 years ago
- Common stop words in a variety of languages☆24Updated last month
- Pingora reverse proxy example☆69Updated last year
- Server-sent events (SSE) client implementation for Rust☆98Updated 2 weeks ago