samclarke / robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
☆150Updated last week
Related projects ⓘ
Alternatives and complementary repositories for robots-parser
- A cacheable dns.lookup(…) that respects TTL☆193Updated 9 months ago
- Parser for robots.txt for node.js☆66Updated 3 years ago
- Tiny LRU cache for Client or Server☆156Updated this week
- Get all urls in a string☆365Updated last year
- A ready to use http and https agent for working with proxies that keeps connections alive!☆182Updated last year
- A stupidly simple key/value storage using files to persist the data☆165Updated 4 months ago
- JavaScript Library to extract domains, subdomains and public suffixes from complex URIs.☆483Updated this week
- Convert a string/promise/array/iterable/asynciterable/buffer/typedarray/arraybuffer/object into a stream☆207Updated 11 months ago
- Get a stream as a string, Buffer, ArrayBuffer or array☆341Updated last week
- 🕐 Performance timings for HTTP requests☆192Updated 3 years ago
- 🇫🇷 NodeJS language detection library using n-gram☆399Updated 4 years ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆231Updated last month
- Short links expander for node.js☆116Updated last year
- parses sitemaps in Node.JS☆112Updated 2 weeks ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆148Updated 3 months ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆508Updated last month
- Get a random temporary file or directory path☆424Updated last year
- Less memory usage version of geoip-lite by supporting only country lookup.☆89Updated this week
- It parses the accept-language header from an HTTP request☆163Updated 2 years ago
- Run some code when the process exits☆282Updated last year
- Run promise-returning & async functions concurrently with optional limited concurrency☆314Updated last year
- Clean up error stack traces☆337Updated last year
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.☆315Updated 2 months ago
- Maxmind's GeoLite2 Free Databases Helper☆62Updated last week
- Word-wrapping for javascript.☆158Updated last year
- Sort an array on an object property☆199Updated 2 months ago
- Parse and stringify the HTML <img> srcset attribute☆137Updated 9 months ago
- Gracefully terminates HTTP(S) server.☆324Updated last year
- Node.js framework agnostic library that enables you to forward an http request to another HTTP server. Supported protocols: HTTP, HTTPS, …☆162Updated 9 months ago