samclarke / robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
☆153Updated 4 months ago
Alternatives and similar repositories for robots-parser:
Users that are interested in robots-parser are comparing it to the libraries listed below
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆511Updated last month
- A cacheable dns.lookup(…) that respects TTL☆199Updated last year
- Run promise-returning & async functions concurrently with optional limited concurrency☆326Updated 3 months ago
- Tiny LRU cache for Client or Server☆159Updated this week
- Get a stream as a string, Buffer, ArrayBuffer or array☆345Updated 4 months ago
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆206Updated last year
- Parser for robots.txt for node.js☆67Updated 3 years ago
- A library for colorizing JSON strings☆85Updated 10 months ago
- It parses the accept-language header from an HTTP request☆165Updated 3 years ago
- Rate limit made simple, easy, async.☆323Updated 10 months ago
- Wait for a condition to be true☆158Updated last year
- A queue with custom concurrency and time limit.☆196Updated 2 years ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆162Updated 3 months ago
- A super-fast, promise based cache that reads and writes to the file-system.☆63Updated 7 months ago
- Character encoding detection tool for NodeJS☆285Updated 3 weeks ago
- Get all urls in a string☆367Updated last year
- ☆89Updated last month
- A Node.js module for the optimized JavaScript implementation of the MurmurHash algorithms.☆96Updated last year
- Word-wrapping for javascript.☆161Updated 4 months ago
- encode & decode HTML & XML entities with ease & speed☆332Updated this week
- A small EXIF image metadata reader☆159Updated 2 months ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆244Updated 2 months ago
- URL-safe base64 UUID encoder for generating 22 character slugs☆92Updated 2 months ago
- Simple benchmark for diffrent htmlparsers using real-life data☆102Updated last month
- A zero-dependency cron parser and scheduler for Node.js, Deno and the browser.☆188Updated last week
- Logger-agnostic wrapper that normalizes logs regardless of arg style. Great for large dev teams, old/new projects, and works w/Pino, Bun…☆50Updated 3 months ago
- A tiny (215B) and fast utility to expand a flattened object☆202Updated last year
- fastify plugin to forward the current http request to another server☆154Updated last week
- Gracefully terminates HTTP(S) server.☆327Updated last year
- Parser for RFC2822 (Header) format email addresses☆54Updated 2 months ago