aichat-bot / crawlyLinks
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
☆26Updated last year
Alternatives and similar repositories for crawly
Users that are interested in crawly are comparing it to the libraries listed below
Sorting:
- A Rust crate for extracting readable content from web pages.☆194Updated 2 weeks ago
- Streaming JSON reader and writer written in Rust☆91Updated last week
- in memory Rust database to query your data like a Venn diagram☆130Updated 2 months ago
- A very fast Key-Value store in pure rust☆219Updated 3 weeks ago
- Fake data generator library with 130+ functions written in Rust☆184Updated last month
- ⏱️ An asynchronous task scheduling library written in Rust☆83Updated last month
- 🦀︱High-performance learned index structures☆104Updated last month
- Background jobs inspector for Apalis☆55Updated last week
- Goku is an HTTP load testing application written in Rust☆145Updated 2 months ago
- Self growing / shrinking ThreadPool implementation based on crossbeam's multi-producer multi-consumer channels that enables awaiting the …☆49Updated last year
- A token-based rate limiter based on the leaky bucket algorithm.☆113Updated 3 months ago
- A software framework and runtime for building scalable, connected applications☆57Updated this week
- Generate client libraries that are featureful, human, well-documented, and async based on OpenAPI specs☆95Updated last year
- Task runner and process manager for Rust☆58Updated 2 weeks ago
- A safe sync/async multi-producer, multi-consumer channel☆208Updated 9 months ago
- A middleware for actix-web that provides rate-limiting backed by governor.☆126Updated 3 months ago
- Reports when tokio runtime threads are blocking☆66Updated last year
- A task orchestrator using redis, written in rust☆72Updated 2 years ago
- Include your assets statically into your application with a strict structure☆93Updated 2 months ago
- Simple string matching with single- and multiple-wildcard operator☆94Updated 2 months ago
- A new type of asynchronous database combining the concepts of distributed databases and blockchain☆112Updated last week
- Rust PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.☆65Updated last year
- Common stop words in a variety of languages☆25Updated 5 months ago
- Rust web servers without async/await.☆221Updated last year
- Structures and tools to parse, navigate and validate OpenAPI v3.1 specifications.☆69Updated last week
- Tako is a lightweight and minimalistic web framework built on Tokio and Hyper written in Rust.☆142Updated 2 weeks ago
- Parallelo Parallel Library (PPL) is a small parallel framework that brings Structured Parallel Programming in Rust.☆79Updated 5 months ago
- Just like `serde`, but with more exhaustive error reporting capabilities. Designed for public API payloads.☆220Updated this week
- Rust bindings to the Vosk API Speech Recognition library☆166Updated 6 months ago
- 🌿 SQL schema definition and discovery☆219Updated last month