samclarke / robots-parserLinks
NodeJS robots.txt parser with support for wildcard (*) matching.
☆156Updated 7 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆512Updated 5 months ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆251Updated last week
- Parse through any sitemap in Node.js☆120Updated 3 weeks ago
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆329Updated 3 months ago
- Parser for robots.txt for node.js☆67Updated 4 years ago
- Normalize + and . emails -- ex.ample+foobar@gmail.com => example@gmail.com☆42Updated 5 years ago
- Get all urls in a string☆369Updated last year
- Tiny LRU cache for Client or Server☆162Updated this week
- 🇫🇷 NodeJS language detection library using n-gram☆402Updated 4 years ago
- A super-fast, promise based cache that reads and writes to the file-system.☆65Updated 11 months ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆168Updated 6 months ago
- A cacheable dns.lookup(…) that respects TTL☆208Updated last year
- It parses the accept-language header from an HTTP request☆166Updated 3 years ago
- Run promise-returning & async functions concurrently with optional limited concurrency☆338Updated 6 months ago
- Word-wrapping for javascript.☆160Updated 7 months ago
- A fast, safe, compliant XML parser for Node.js and browsers.☆310Updated 7 months ago
- Fast Porter stemmer implementation☆132Updated 2 years ago
- Detect possibly catastrophic, exponential-time regular expressions☆184Updated 2 years ago
- An Express middleware for quick'n'easy server-sent events.☆124Updated 2 months ago
- This module adds [Server-Timing](https://www.w3.org/TR/server-timing/) to response headers, see [example](https://server-timing.now.sh/) …☆126Updated last week
- NPM open registry☆68Updated this week
- A library for colorizing JSON strings☆86Updated last year
- Memoize promise-returning & async functions☆407Updated 2 years ago
- Timeout a promise after a specified amount of time☆291Updated 5 months ago
- A zero-dependency cron parser and scheduler for Node.js, Deno and the browser.☆190Updated last week
- Dependency-free stream utils for Node.js☆102Updated 2 years ago
- Safely and quickly serialize JavaScript objects☆351Updated 6 months ago
- Streaming line delimited json parser + serializer☆119Updated 3 years ago
- Execute a function on exit without leaking memory, allowing all objects to be garbage collected☆146Updated last year
- Throttles arbitrary code to execute a maximum number of times per interval. Best for making throttled API requests.☆123Updated 2 months ago