samclarke / robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
☆153Updated 4 months ago
Alternatives and similar repositories for robots-parser:
Users that are interested in robots-parser are comparing it to the libraries listed below
- Run promise-returning & async functions concurrently with optional limited concurrency☆326Updated 2 months ago
- A module for node.js and the browser that takes in text and strips it of stopwords☆243Updated 2 months ago
- A cacheable dns.lookup(…) that respects TTL☆199Updated last year
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆511Updated last month
- Parser for robots.txt for node.js☆67Updated 3 years ago
- Node.js DNS over HTTPS - Tangerine is the best drop-in replacement for dns.promises.Resolver using DNS over HTTPS ("DoH") via undici wit…☆161Updated 2 months ago
- A Node.js module for the optimized JavaScript implementation of the MurmurHash algorithms.☆96Updated last year
- Get all urls in a string☆367Updated last year
- A super-fast, promise based cache that reads and writes to the file-system.☆63Updated 7 months ago
- Run some code when the process exits☆282Updated last year
- 🇫🇷 NodeJS language detection library using n-gram☆400Updated 4 years ago
- JSON.parse() drop-in replacement with prototype poisoning protection☆173Updated 4 months ago
- making fetch happen for npm☆192Updated 3 months ago
- JSON.parse() drop-in replacement with prototype poisoning protection☆229Updated this week
- Tiny LRU cache for Client or Server☆159Updated this week
- Short links expander for node.js☆120Updated last year
- Get a stream as a string, Buffer, ArrayBuffer or array☆345Updated 4 months ago
- A small EXIF image metadata reader☆159Updated last month
- 🌌 Tiniest body parser in the universe. Built for modern Node.js☆195Updated 2 weeks ago
- NPM open registry☆68Updated 2 years ago
- Useful utilities for working with Uint8Array (and Buffer)☆251Updated last month
- Parse and stringify the HTML <img> srcset attribute☆138Updated last year
- The fastest way to parse JSON safely☆88Updated 3 years ago
- Zero dependencies, gracefully terminates HTTP(S) server.☆18Updated last year
- A ready to use http and https agent for working with proxies that keeps connections alive!☆185Updated last year
- Detects if a file is binary in Node.js. Similar to Perl's -B☆164Updated 4 months ago
- Make a synchronous function asynchronous by running it in a worker☆249Updated 5 months ago
- A fast, safe, compliant XML parser for Node.js and browsers.☆305Updated 4 months ago
- Dependency-free RFC 3986 URI toolbox☆91Updated this week
- Maxmind's GeoLite2 Free Databases Helper☆63Updated 2 months ago