brendonboshell / supercrawlerLinks
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
☆379Updated 2 years ago
Alternatives and similar repositories for supercrawler
Users that are interested in supercrawler are comparing it to the libraries listed below
Sorting:
- Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.☆434Updated 2 years ago
- Email automation driven by headless chrome.☆168Updated 4 years ago
- Javascript scraping module based on puppeteer for many different search engines...☆561Updated 2 years ago
- Google Search SERP Scraper☆117Updated last week
- Web crawler for Node.JS☆255Updated 7 years ago
- House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.☆128Updated 2 years ago
- Blazingly fast, multi tenant, faceted search API☆312Updated 4 years ago
- Node.js email SMTP verification, powered by EmailChecker.com API☆294Updated last week
- Chromium / Puppeteer site crawler☆49Updated 5 years ago
- Cloud crawler functions for scrapeulous☆45Updated 4 years ago
- Nodejs lib to parse Google SERP html pages☆47Updated 2 years ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆70Updated 4 years ago
- plugin to extract keywords and key-phrases☆334Updated last year
- Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.☆345Updated 7 years ago
- Library and CLI for automating captcha verification across multiple providers.☆123Updated 5 years ago
- Verify email address checking MX records, and SMTP connection.☆124Updated 4 years ago
- Node module that summarizes text using a naive summarization algorithm☆770Updated last year
- Automatically extracts structured information from webpages☆109Updated 3 years ago
- A look at how LinkedIn spies on its users.☆832Updated 6 years ago
- Simple, lightweight and expressive web scraping with Node.js☆153Updated 4 years ago
- High-performance FlexSearch Server for Node.js (Cluster)☆189Updated 6 years ago
- A light, fast and flexible javascript tracking library☆262Updated 2 years ago
- A Better Scraper, with Puppeteer☆43Updated 2 months ago
- Easily create XML sitemaps for your website.☆442Updated last year
- A Node.js module to search and scrape Google.☆456Updated 7 years ago
- A JS lib that provides a mechanism to get/set cookies that can be shared across domains☆164Updated 5 years ago
- Google search scraper with captcha solving support☆91Updated 6 years ago
- An AliExpress spider for Node☆45Updated 8 years ago
- Nodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Support…☆112Updated 2 years ago
- keywords-extract - Command line tool extract keywords from any web page.☆61Updated 7 years ago