samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆157Updated 9 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- A fast, safe, compliant XML parser for Node.js and browsers.☆313Updated 10 months ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆254Updated 2 months ago
- A high-performance, lightweight LRU cache. Built for developers who need fast caching without compromising on features.☆168Updated this week
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆512Updated 2 weeks ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆169Updated 8 months ago
- Streaming line delimited json parser + serializer☆123Updated 3 years ago
- A cacheable dns.lookup(…) that respects TTL☆211Updated last year
- A library for colorizing JSON strings☆86Updated last year
- Obscene words filter for nanoid☆88Updated 2 years ago
- Run promise-returning & async functions concurrently with optional limited concurrency☆338Updated 8 months ago
- Get all urls in a string☆368Updated 2 years ago
- Rate limit made simple, easy, async.☆333Updated last week
- a monorepo of npm packages☆201Updated last month
- Get a stream as a string, Buffer, ArrayBuffer or array☆352Updated 9 months ago
- This module adds [Server-Timing](https://www.w3.org/TR/server-timing/) to response headers, see [example](https://server-timing.now.sh/) …☆126Updated last week
- making fetch happen for npm☆201Updated this week
- Chain functions as transform streams.☆94Updated 3 months ago
- Normalize + and . emails -- ex.ample+foobar@gmail.com => example@gmail.com☆42Updated 5 years ago
- It parses the accept-language header from an HTTP request☆166Updated 3 years ago
- JSON.parse() drop-in replacement with prototype poisoning protection☆237Updated 2 months ago
- Slug generator with less than 1 KB and no dependencies, RFC 3986 compliant☆89Updated 2 years ago
- Convert number to a human readable string: `13500` → `13.5K`☆32Updated last week
- Parse through any sitemap in Node.js☆122Updated last month
- 🚸 Cross platform children list of a PID.☆126Updated 2 years ago
- A small EXIF image metadata reader☆169Updated 7 months ago
- 🇫🇷 NodeJS language detection library using n-gram☆405Updated 5 years ago
- ☆94Updated 6 months ago
- *Maintained fork of Later.* A javascript library for defining recurring schedules and calculating future (or past) occurrences for them. …☆145Updated last year
- Convert Unicode characters to Latin characters using transliteration☆305Updated 2 years ago
- Parse and stringify the HTML <img> srcset attribute☆140Updated last year