samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆159Updated 10 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- A fast, safe, compliant XML parser for Node.js and browsers.☆312Updated 10 months ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆254Updated 3 months ago
- A cacheable dns.lookup(…) that respects TTL☆214Updated last year
- Obscene words filter for nanoid☆90Updated 2 years ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆169Updated 8 months ago
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆221Updated 3 years ago
- A high-performance, lightweight LRU cache. Built for developers who need fast caching without compromising on features.☆168Updated last week
- Streaming line delimited json parser + serializer☆123Updated 4 years ago
- Run promise-returning & async functions concurrently with optional limited concurrency☆339Updated 8 months ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆512Updated this week
- Rate limit made simple, easy, async.☆334Updated last week
- A library for colorizing JSON strings☆86Updated last year
- a monorepo of npm packages☆202Updated last month
- A queue with custom concurrency and time limit.☆196Updated 3 years ago
- Parser for RFC2822 (Header) format email addresses☆54Updated 2 months ago
- ☆94Updated 6 months ago
- making fetch happen for npm☆200Updated 3 weeks ago
- NPM open registry☆68Updated this week
- Character encoding detection tool for NodeJS☆294Updated 2 months ago
- Get all urls in a string☆368Updated 2 years ago
- Preserves types over JSON, BSON or socket.io☆71Updated 3 weeks ago
- JSON.parse() drop-in replacement with prototype poisoning protection☆237Updated last week
- Check where your code is running. In the browser or in node.js environment.☆251Updated 10 months ago
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆208Updated last year
- Unabashedly-barebones memoization library with an aim toward speed☆121Updated last month
- This module adds [Server-Timing](https://www.w3.org/TR/server-timing/) to response headers, see [example](https://server-timing.now.sh/) …☆126Updated this week
- Drop in replacement for the Node.js fs library backed by AWS S3☆126Updated last year
- convert a human readable duration to ms☆339Updated 5 months ago
- Wait for a condition to be true☆160Updated 2 years ago
- Chain functions as transform streams.☆94Updated 3 months ago