samclarke / robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
β154Updated 6 months ago
Alternatives and similar repositories for robots-parser
Users that are interested in robots-parser are comparing it to the libraries listed below
Sorting:
- π«π· NodeJS language detection library using n-gramβ402Updated 4 years ago
- Run promise-returning & async functions concurrently with optional limited concurrencyβ332Updated 4 months ago
- Tiny LRU cache for Client or Serverβ163Updated last week
- Parser for robots.txt for node.jsβ67Updated 4 years ago
- A library for colorizing JSON stringsβ85Updated last year
- NPM open registryβ68Updated 2 years ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.β511Updated 3 months ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici witβ¦β164Updated 4 months ago
- Streaming line delimited json parser + serializerβ119Updated 3 years ago
- A queue with custom concurrency and time limit.β197Updated 3 years ago
- JSON.parse() drop-in replacement with prototype poisoning protectionβ231Updated this week
- A cacheable dns.lookup(β¦) that respects TTLβ204Updated last year
- Run some code when the process exitsβ286Updated last year
- Get all urls in a stringβ369Updated last year
- Language detection for Javascript (Node). Based on the CLD2 (Compact Language Detector) library from Google.β326Updated 2 months ago
- Human readable time distances for javascriptβ465Updated last year
- Node.js framework agnostic library that enables you to forward an http request to another HTTP server. Supported protocols: HTTP, HTTPS, β¦β162Updated last year
- A fast, safe, compliant XML parser for Node.js and browsers.β308Updated 6 months ago
- A Node.js module for the optimized JavaScript implementation of the MurmurHash algorithms.β97Updated last year
- Maxmind's GeoLite2 Free Databases Helperβ63Updated last week
- encode & decode HTML & XML entities with ease & speedβ338Updated this week
- Creates a cryptographically strong UIDβ256Updated 11 months ago
- Node.js implementation of HighwayHash, Google's fast and strong hash functionβ221Updated 3 years ago
- ASCII transliterations of Unicode textβ148Updated 7 months ago
- Parser for RFC2822 (Header) format email addressesβ54Updated 4 months ago
- Execute a function on exit without leaking memory, allowing all objects to be garbage collectedβ145Updated last year
- π Performance timings for HTTP requestsβ191Updated 3 years ago
- Less memory usage version of geoip-lite by supporting only country lookup.β92Updated this week
- π³π΄ Β Lookup information with ISO 3166-1 alpha-2, ISO 3166-1 alpha-3 and ISO 3166-1 numericβ58Updated last year
- High performance Base64 encoding and decoding for Node.js using SIMD (AVX2, AVX, SSE4.2, SSE4.1, SSSE3) accelerationβ108Updated 3 years ago