samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆165Updated last month
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- A fast, safe, compliant XML parser for Node.js and browsers.☆316Updated last year
- A cacheable dns.lookup(…) that respects TTL☆216Updated last year
- A module for node.js and the browser that takes in text and strips it of stopwords☆259Updated 2 months ago
- Obscene words filter for nanoid☆90Updated 2 years ago
- Run promise-returning & async functions concurrently with optional limited concurrency☆344Updated 3 months ago
- A high-performance, lightweight LRU cache. Built for developers who need fast caching without compromising on features.☆176Updated last week
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆174Updated this week
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆514Updated last month
- A library for colorizing JSON strings☆88Updated last year
- Streaming line delimited json parser + serializer☆128Updated 4 years ago
- 🇫🇷 NodeJS language detection library using n-gram☆408Updated 5 years ago
- a monorepo of npm packages☆207Updated last month
- Safely and quickly serialize JavaScript objects☆356Updated last year
- Straightforward HTML parser for JavaScript☆142Updated last month
- Rate limit made simple, easy, async.☆336Updated last month
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆221Updated 4 years ago
- A ready to use http and https agent for working with proxies that keeps connections alive!☆190Updated 2 years ago
- Character encoding detection tool for NodeJS☆300Updated 3 weeks ago
- Run some code when the process exits☆298Updated last month
- ☆94Updated 10 months ago
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆212Updated 3 months ago
- JavaScript domain name parser based on the Public Suffix List☆429Updated 8 months ago
- Chain functions as transform streams.☆95Updated 7 months ago
- making fetch happen for npm☆204Updated last month
- A stream implementation that does more by doing less☆255Updated 2 months ago
- Check if servers are reachable☆382Updated 3 months ago
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆334Updated 9 months ago
- A promises port of https://github.com/raszi/node-tmp☆133Updated last year
- Generate self-signed certificates from node.js☆250Updated last week
- Dependency-free stream utils for Node.js☆106Updated 2 years ago