aichat-bot / crawly
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
☆21Updated 7 months ago
Alternatives and similar repositories for crawly:
Users that are interested in crawly are comparing it to the libraries listed below
- Rust interface for the WebRTC Voice-Activity-Module☆26Updated 4 years ago
- End-to-end encryption sdk with user management.☆45Updated last week
- Common stop words in a variety of languages☆19Updated 2 weeks ago
- Distributed locks with Redis, implemented in Rust☆55Updated 3 months ago
- A token-based rate limiter based on the leaky bucket algorithm.☆105Updated 5 months ago
- Load testing framework in Rust☆114Updated 3 weeks ago
- A problem type to be returned by HTTP APIs☆56Updated last week
- A generic abstraction of paginated APIs☆59Updated last year
- A Queue Abstraction Layer for Rust (Redis, RabbitMQ, SQS, etc.)☆64Updated last week
- General purpose OAuth2 component for Yew☆45Updated 9 months ago
- Framework to fuzz Rust/Tauri applications based on behavior policies☆47Updated last month
- RSMQ port to async rust. RSMQ is a simple redis queue system that works in any redis v2.6+☆45Updated 3 weeks ago
- A very fast Key-Value store in pure rust☆94Updated this week
- Comparing performance-oriented string-processing libraries for substring search, multi-pattern matching, hashing, and Levenshtein edit-di…☆45Updated last month
- Common Rust library☆32Updated this week
- fast ip -> country lookup using asn database☆72Updated this week
- Simple string matching with single- and multiple-wildcard operator☆80Updated 3 months ago
- `acts` is a fast, tiny, extensiable workflow engine☆35Updated last month
- Opinionated framework for web services☆30Updated last year
- Simple pure Rust AWS S3 Client following a Sans-IO approach☆115Updated 2 weeks ago
- A safe sync/async multi-producer, multi-consumer channel☆199Updated 2 months ago
- Graceful shutdown util for Rust projects using the Tokio Async runtime.☆126Updated 3 months ago
- Concurrent and multi-stage data ingestion and data processing with Rust+Tokio☆46Updated 2 years ago
- E-mail delivery library for Rust with DKIM support☆214Updated 5 months ago
- Your AI Copilot in Rust☆46Updated last year
- A tracing layer for shipping logs to Grafana Loki☆51Updated this week
- Safe asynchronous access to Diesel and the bb8 connection manager☆14Updated this week
- An SMTP server for test and development environments written in Rust☆177Updated 5 months ago
- A minimal implementation of consistent hashing☆54Updated 5 months ago
- Prost Well-Known-Types serialization and deserialization.☆77Updated 5 months ago