samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆159Updated 11 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- A fast, safe, compliant XML parser for Node.js and browsers.☆313Updated 11 months ago
- A cacheable dns.lookup(…) that respects TTL☆214Updated last year
- A high-performance, lightweight LRU cache. Built for developers who need fast caching without compromising on features.☆168Updated this week
- A module for node.js and the browser that takes in text and strips it of stopwords☆255Updated 3 months ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆512Updated 3 weeks ago
- Streaming line delimited json parser + serializer☆123Updated 4 years ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆170Updated 9 months ago
- Obscene words filter for nanoid☆90Updated 2 years ago
- A library for colorizing JSON strings☆86Updated last year
- 🇫🇷 NodeJS language detection library using n-gram☆405Updated 5 years ago
- ☆94Updated 7 months ago
- a monorepo of npm packages☆204Updated 2 months ago
- Run promise-returning & async functions concurrently with optional limited concurrency☆341Updated 2 weeks ago
- Rate limit made simple, easy, async.☆335Updated last month
- Check where your code is running. In the browser or in node.js environment.☆251Updated 10 months ago
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆332Updated 7 months ago
- Straightforward HTML parser for JavaScript☆142Updated last year
- Parse through any sitemap in Node.js☆124Updated 2 months ago
- Timeout a promise after a specified amount of time☆293Updated 3 weeks ago
- ☆200Updated last year
- Drop in replacement for the Node.js fs library backed by AWS S3☆126Updated last year
- Gracefully terminates HTTP(S) server.☆333Updated 2 years ago
- Get all urls in a string☆368Updated 2 years ago
- 🌌 Tiniest body parser in the universe. Built for modern Node.js☆200Updated 3 weeks ago
- Node.js framework agnostic library that enables you to forward an http request to another HTTP server. Supported protocols: HTTP, HTTPS, …☆162Updated last year
- Server sent events for NodeJS☆72Updated 2 years ago
- JSON.parse() drop-in replacement with prototype poisoning protection☆237Updated this week
- URL-safe base64 UUID encoder for generating 22 character slugs☆93Updated 9 months ago
- Format a date using Intl.DateTimeFormat goodness.☆61Updated 2 years ago
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆221Updated 3 years ago