samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆156Updated 7 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆511Updated 4 months ago
- A cacheable dns.lookup(…) that respects TTL☆204Updated last year
- Parser for robots.txt for node.js☆67Updated 4 years ago
- Parse through any sitemap in Node.js☆119Updated this week
- A module for node.js and the browser that takes in text and strips it of stopwords☆250Updated 5 months ago
- A ready to use http and https agent for working with proxies that keeps connections alive!☆187Updated last year
- Decorator for a `fetch` function to support automatic cookie storage and population. 🍪☆142Updated 5 months ago
- A Node.js module for the optimized JavaScript implementation of the MurmurHash algorithms.☆98Updated last year
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆208Updated last year
- Timeout a promise after a specified amount of time☆290Updated 5 months ago
- ☆92Updated 3 months ago
- Less memory usage version of geoip-lite by supporting only country lookup.☆92Updated this week
- A library for colorizing JSON strings☆86Updated last year
- a monorepo of npm packages☆199Updated 3 months ago
- Tiny LRU cache for Client or Server☆163Updated this week
- A fast, safe, compliant XML parser for Node.js and browsers.☆311Updated 7 months ago
- Parse and stringify the HTML <img> srcset attribute☆140Updated last year
- Slug generator with less than 1 KB and no dependencies, RFC 3986 compliant☆89Updated last year
- Rate limit made simple, easy, async.☆332Updated last month
- Straightforward HTML parser for JavaScript☆139Updated 10 months ago
- Get all urls in a string☆369Updated last year
- Character encoding detection tool for NodeJS☆291Updated 3 months ago
- Node.js implementation of HighwayHash, Google's fast and strong hash function☆221Updated 3 years ago
- encode & decode HTML & XML entities with ease & speed☆339Updated this week
- JSON.parse() drop-in replacement with prototype poisoning protection☆233Updated 3 weeks ago
- Obscene words filter for nanoid☆88Updated 2 years ago
- Maxmind's GeoLite2 Free Databases Helper☆63Updated this week
- Logger-agnostic wrapper that normalizes logs regardless of arg style. Great for large dev teams, old/new projects, and works w/Pino, Bun…☆51Updated 6 months ago
- Get a stream as a string, Buffer, ArrayBuffer or array☆349Updated 7 months ago
- Clean up error stack traces☆340Updated 2 years ago