Curated list of common web crawler and bot User-Agent strings used across the internet.
☆29Aug 3, 2026Updated last month
Alternatives and similar repositories for top-crawler-agents
Users that are interested in top-crawler-agents are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Setup HTTPS details related to TLS according to the User Agent provided.☆16Aug 3, 2026Updated last month
- Run official FLAC tools "flac" and "metaflac" as WebAssembly, on browsers or Deno.☆18Nov 29, 2024Updated last year
- ☆24Apr 4, 2026Updated 5 months ago
- ☆10Nov 26, 2022Updated 3 years ago
- Annotate functions as pure for Rollup, Vite, esbuild, and rspack☆13Dec 15, 2025Updated 8 months ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Build time stat reporting for unplugin☆13Updated this week
- Format a stream to make it more human readable☆16Oct 2, 2014Updated 11 years ago
- The program can be used to scrape the content from an article from web by an input of a set of URLs in a text file or a URL. This project…☆18Aug 5, 2020Updated 6 years ago
- 💍 Pick out specific elements from the component <slot>☆12Oct 11, 2025Updated 10 months ago
- a CLI that synchronizes two folders with minimal writes☆13Feb 25, 2022Updated 4 years ago
- badapi.net - a badly behaved REST API☆17Jun 23, 2016Updated 10 years ago
- Upgrade the maximum file descriptor number ('nofile') that can be opened by this process☆18Jul 17, 2020Updated 6 years ago
- Testing various algorithms for resolving node overlaps in a flow☆16Dec 1, 2025Updated 9 months ago
- ☆15Sep 21, 2019Updated 6 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- A simplified FinalizationRegistry utility that works.☆20Aug 1, 2024Updated 2 years ago
- Promise-based pixiv cookie getter☆11Mar 4, 2017Updated 9 years ago
- Fastest, plain JavaScript-based, regexp escaping library for JavaScript, works in both Node.js and browser.☆46Updated this week
- A Markdown formatter for dox. Takes dox's JSON output as stdin and writes Markdown to stdout.☆13Jan 16, 2017Updated 9 years ago
- 🔐 A modern & lightweight TLS library & CLI. Built for automation—HTTPS by default. Similar to mkcert.☆47Updated this week
- Analyzes target website for anti-scraping protections and performance. Saves screenshots/HTML snapshots.☆12Aug 13, 2025Updated last year
- A simple tool that allows you to add and remove any querystring from the url☆10Jan 4, 2023Updated 3 years ago
- 💻 A minimalistic CLI to bootstrap projects with different frameworks☆14Dec 15, 2023Updated 2 years ago
- Web page preview and analysis tool☆12Jan 11, 2023Updated 3 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- A hook for triggering the Web Share API (with a fallback)☆15Jan 6, 2023Updated 3 years ago
- Apple OpenSource download tool☆14Apr 17, 2020Updated 6 years ago
- An implementation of window.fetch in Node.js using Minipass streams☆61Aug 3, 2026Updated last month
- Check whether a binary version satisfies a semver range☆23Apr 3, 2024Updated 2 years ago
- Visual search interface☆11Nov 30, 2021Updated 4 years ago
- ∞ An icon library plugin for Penpot.☆15Nov 23, 2024Updated last year
- ⚙️ IntelliSense integration for Pinceau features☆16Feb 13, 2023Updated 3 years ago
- Convert between images, pixel buffers and textures.☆15Aug 29, 2026Updated last week
- A simple menu bar app for macOS that displays a calendar.☆13Mar 1, 2026Updated 6 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Track SVG Splash feature requests and discussions☆13Apr 23, 2020Updated 6 years ago
- Set up custom icons for your macOS Applications without fear that an auto-update will scrap your changes☆13Sep 28, 2016Updated 9 years ago
- Codable Color Struct☆13Jun 20, 2026Updated 2 months ago
- Opinionated CLI to generate GitHub Sponsors svg. Credit @antfu☆23Feb 25, 2022Updated 4 years ago
- An airflow deployment configuration with sane defaults☆10Jun 6, 2019Updated 7 years ago
- Generate custom Mac OS folder icons with a desired image as stamp☆12Oct 3, 2023Updated 2 years ago
- ☆17Jun 8, 2021Updated 5 years ago