samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆157Updated 9 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- Run promise-returning & async functions concurrently with optional limited concurrency☆339Updated 7 months ago
- A fast, safe, compliant XML parser for Node.js and browsers.☆312Updated 9 months ago
- A cacheable dns.lookup(…) that respects TTL☆211Updated last year
- Streaming line delimited json parser + serializer☆122Updated 3 years ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆512Updated last week
- A module for node.js and the browser that takes in text and strips it of stopwords☆253Updated last month
- Tiny LRU cache for Client or Server☆164Updated last week
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆170Updated 7 months ago
- A library for colorizing JSON strings☆86Updated last year
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆220Updated 3 years ago
- Rate limit made simple, easy, async.☆333Updated last month
- a monorepo of npm packages☆200Updated 2 weeks ago
- Convert Unicode characters to Latin characters using transliteration☆304Updated 2 years ago
- 🇫🇷 NodeJS language detection library using n-gram☆406Updated 5 years ago
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆208Updated last year
- Less memory usage version of geoip-lite by supporting only country lookup.☆91Updated this week
- Straightforward HTML parser for JavaScript☆139Updated last year
- Normalize + and . emails -- ex.ample+foobar@gmail.com => example@gmail.com☆42Updated 5 years ago
- Obscene words filter for nanoid☆88Updated 2 years ago
- Node.js framework agnostic library that enables you to forward an http request to another HTTP server. Supported protocols: HTTP, HTTPS, …☆162Updated last year
- JSON.parse() drop-in replacement with prototype poisoning protection☆237Updated last month
- Parser for RFC2822 (Header) format email addresses☆54Updated last month
- ☆94Updated 5 months ago
- Chain functions as transform streams.☆93Updated 2 months ago
- Get all urls in a string☆368Updated last year
- Spec-compliant FormData implementation for Node.js☆146Updated 2 months ago
- Generate self-signed certificates from node.js☆244Updated last week
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆333Updated 5 months ago
- Get a stream as a string, Buffer, ArrayBuffer or array☆352Updated 9 months ago
- Run some code when the process exits☆292Updated last year