samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆165Updated 2 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- A cacheable dns.lookup(…) that respects TTL☆217Updated 2 years ago
- A fast, safe, compliant XML parser for Node.js and browsers.☆316Updated last year
- A module for node.js and the browser that takes in text and strips it of stopwords☆261Updated 2 weeks ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆181Updated this week
- A library for colorizing JSON strings☆90Updated last year
- Run promise-returning & async functions concurrently with optional limited concurrency☆344Updated 4 months ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆514Updated last week
- A high-performance, lightweight LRU cache. Built for developers who need fast caching without compromising on features.☆177Updated this week
- JSON.parse() drop-in replacement with prototype poisoning protection☆240Updated 2 months ago
- Rate limit made simple, easy, async.☆336Updated 2 months ago
- Node.js framework agnostic library that enables you to forward an http request to another HTTP server. Supported protocols: HTTP, HTTPS, …☆162Updated 2 years ago
- a monorepo of npm packages☆207Updated 2 weeks ago
- Character encoding detection tool for NodeJS☆300Updated 2 months ago
- NPM open registry☆73Updated 2 months ago
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆335Updated 11 months ago
- A queue with custom concurrency and time limit.☆196Updated 3 years ago
- 🌌 Tiniest body parser in the universe. Built for modern Node.js☆202Updated 2 months ago
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆221Updated 4 years ago
- 🇫🇷 NodeJS language detection library using n-gram☆410Updated 5 years ago
- Streaming line delimited json parser + serializer☆131Updated 4 years ago
- Word-wrapping for javascript.☆161Updated 3 months ago
- A light statistical package that operates on Arrays.☆74Updated 5 years ago
- making fetch happen for npm☆205Updated 2 months ago
- Obscene words filter for nanoid☆91Updated 3 years ago
- Detects if a file is binary in Node.js. Similar to Perl's -B☆171Updated 3 weeks ago
- Drop in replacement for the Node.js fs library backed by AWS S3☆132Updated 2 years ago
- Timeout a promise after a specified amount of time☆299Updated 3 months ago
- Get a stream as a string, Buffer, ArrayBuffer or array☆358Updated last year
- Unabashedly-barebones memoization library with an aim toward speed☆120Updated 5 months ago
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆213Updated 4 months ago