monperrus / crawler-user-agents
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome
β1,226Updated last week
Alternatives and similar repositories for crawler-user-agents:
Users that are interested in crawler-user-agents are comparing it to the libraries listed below
- π€/π¨βπ¦° Detect bots/crawlers/spiders using the user agent stringβ984Updated this week
- Cross-language temporary (disposable/throwaway) email detection library. Covers 55 734+ fake email providers.β1,696Updated this week
- β554Updated 11 months ago
- Device information and digital fingerprinting written in pure JavaScript.β2,135Updated 2 years ago
- A list of disposable email domainsβ1,324Updated last year
- The regex file necessary to build language ports of Browserscope's user agent parser.β768Updated 3 weeks ago
- A list of disposable/temporary email address domainsβ1,030Updated last week
- Fingerprinting script of Fingerprint-Scannerβ244Updated 11 months ago
- Bot detection library that runs in the browser. Detects automation tools and frameworks. No server required, runs 100% on the client. MITβ¦β1,150Updated this week
- An open source list of ASNs known to belong to cloud, managed hosting, and colo facilities.β366Updated 2 weeks ago
- Allows you to detect the extension AdBlock (and other)β621Updated 5 years ago
- Categorization of IP Addressesβ528Updated 2 years ago
- Automatic PageSpeed optimization module for Nginxβ4,361Updated last year
- Whois server list for all top level domains.β319Updated 4 years ago
- Daily updated repository for https://github.com/disposable/disposableβ475Updated this week
- Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.β649Updated 3 years ago
- The main project repositoryβ433Updated this week
- A JavaScript library for generating random user agents with data that's updated daily.β1,017Updated this week
- Access https://infosimples.github.io/detect-headless to run several headless detection tests against your browser.β279Updated 4 years ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.β510Updated 3 weeks ago
- A list of temporary email providersβ1,094Updated 2 weeks ago
- Luminati HTTP/HTTPS Proxy managerβ754Updated 2 weeks ago
- Extract the minimal CSS used in a set of URLs with puppeteerβ351Updated 2 years ago
- Easily create XML sitemaps for your website.β429Updated 7 months ago
- Maxmind GEO Lookupβ618Updated last week
- Transparency and Consent Framework v1.1 Consent String SDK - javascriptβ90Updated 11 months ago
- Google client for SERPSβ170Updated 9 months ago
- Puppeteer(Chrome headless node API) based web page rendererβ318Updated 4 months ago
- Puppeteer Pool, run a cluster of instances in parallelβ3,324Updated 9 months ago
- Apache module for rewriting web pages to reduce latency and bandwidth.β694Updated last year