monperrus / crawler-user-agentsLinks
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome
☆1,285Updated 2 months ago
Alternatives and similar repositories for crawler-user-agents
Users that are interested in crawler-user-agents are comparing it to the libraries listed below
Sorting:
- 🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string☆1,051Updated last week
- Daily updated repository for https://github.com/disposable/disposable☆514Updated this week
- An open source list of ASNs known to belong to cloud, managed hosting, and colo facilities.☆412Updated 5 months ago
- A list of disposable email domains☆1,348Updated 4 months ago
- The quickest way to run thumbor.☆292Updated 4 months ago
- A list of disposable/temporary email address domains☆1,149Updated 3 weeks ago
- binlist.net data repo☆633Updated last year
- Block bad, possibly even malicious web crawlers (automated bots) using Nginx☆866Updated 4 years ago
- Whois server list for all top level domains.☆325Updated 4 years ago
- Easily create XML sitemaps for your website.☆435Updated last year
- Community-contributed list of referrer spammers. Comment +1 in any issue or Pull request and the spammer will be added to the list!☆674Updated 2 weeks ago
- IP geolocation web server☆736Updated 3 years ago
- reliable fake and temp email filter solution for site operators☆280Updated this week
- Detects ad blockers (AdBlock, ...)☆1,905Updated last year
- Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti…☆4,442Updated this week
- This project aims to modify your nginx configuration to let you get the real ip address of your visitors.☆708Updated last year
- A list of temporary email providers☆1,133Updated 2 months ago
- JavaScript domain name parser based on the Public Suffix List☆418Updated 3 months ago
- This is a project for a browser fingerprinting technique that can track users not only within a single browser but also across different …☆1,268Updated 3 years ago
- Cross-language temporary (disposable/throwaway) email detection library. Covers 55 734+ fake email providers.☆1,773Updated this week
- Maxmind GEO Lookup☆627Updated 2 weeks ago
- NGINX module for Brotli compression☆2,167Updated last year
- Automatic PageSpeed optimization module for Nginx☆4,358Updated 2 years ago
- IP Intelligence is a free Proxy VPN TOR and Bad IP detection tool to prevent Fraud, stolen content, and malicious users. Block proxies, V…☆324Updated 5 months ago
- Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress T…☆904Updated this week
- Device information and digital fingerprinting written in pure JavaScript.☆2,188Updated 2 years ago
- Splits a hostname into subdomains, domain and (effective) top-level domains.☆513Updated 5 months ago
- Search Console Archive store your Search Console (Webmaster Tools) data to exceed the 90 days history in a web SEO tool with search & ana…☆31Updated 5 years ago
- Scrapoxy is a super proxies manager that orchestrates all your proxies into one place, rather than spreading management across multiple s…☆2,309Updated last month
- Rotating TOR proxy with Docker☆1,182Updated last year