aichat-bot / crawlyLinks
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
☆25Updated last year
Alternatives and similar repositories for crawly
Users that are interested in crawly are comparing it to the libraries listed below
Sorting:
- Fake data generator library with 130+ functions written in Rust☆169Updated 6 months ago
- Server-sent events (SSE) client implementation for Rust☆91Updated last week
- in memory Rust database to query your data like a Venn diagram☆110Updated 7 months ago
- JSON Schema based validation tool using serde.☆75Updated 2 months ago
- A Queue Abstraction Layer for Rust (Redis, RabbitMQ, SQS, etc.)☆87Updated last month
- Background jobs inspector for Apalis☆28Updated 7 months ago
- Rust crate to retrieve public IP address and geolocation data. 🦀☆36Updated 2 weeks ago
- Distributed locks with Redis, implemented in Rust☆56Updated 9 months ago
- Framework to fuzz Rust/Tauri applications based on behavior policies☆53Updated 5 months ago
- An SMTP server for test and development environments written in Rust☆185Updated 2 months ago
- Self growing / shrinking ThreadPool implementation based on crossbeam's multi-producer multi-consumer channels that enables awaiting the …☆46Updated last year
- Pre-compiled random real User-Agents. Updated weekly for windows, mac, linux, and android☆33Updated last week
- User Agent Parser for Rust☆48Updated 11 months ago
- ⏱️ An asynchronous task scheduling library written in Rust☆64Updated 2 weeks ago
- A port of sidekiq to rust using tokio☆152Updated 5 months ago
- The ultimate Meilisearch client for your terminal☆40Updated last year
- Rust PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.☆63Updated 7 months ago
- A task orchestrator using redis, written in rust☆71Updated 2 years ago
- Streaming JSON reader and writer written in Rust☆78Updated this week
- Rust interface for the WebRTC Voice-Activity-Module☆30Updated 5 years ago
- Simple string matching with single- and multiple-wildcard operator☆87Updated 9 months ago
- Simple, minimal-dependency HTTP client.☆228Updated 3 weeks ago
- Common stop words in a variety of languages☆21Updated 4 months ago
- A loadable Redis module that implements the token bucket algorithm to do rate limiting as a native command☆50Updated this week
- A token-based rate limiter based on the leaky bucket algorithm.☆111Updated 3 months ago
- A very fast Key-Value store in pure rust☆208Updated 2 weeks ago
- A very fast HTTP Server based on Hyper and Rust☆87Updated last year
- A cron job library for Rust☆21Updated last month
- End-to-end encryption sdk with user management.☆45Updated 2 weeks ago
- Goku is an HTTP load testing application written in Rust☆137Updated last month