brendonboshell / supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
☆376Updated last year
Related projects ⓘ
Alternatives and complementary repositories for supercrawler
- Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.☆415Updated last year
- Javascript scraping module based on puppeteer for many different search engines...☆548Updated last year
- Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)☆501Updated 4 years ago
- Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.☆343Updated 6 years ago
- Declarative DOM extraction expression evaluator. 👨⚕️☆695Updated 4 years ago
- Web crawler for Node.JS☆253Updated 6 years ago
- Node.js implementation of a proxy server (think Squid) with support for SSL, authentication and upstream proxy chaining.☆849Updated 3 weeks ago
- Email automation driven by headless chrome.☆164Updated 3 years ago
- Library and CLI for automating captcha verification across multiple providers.☆122Updated 4 years ago
- Automatically extract body content (and other cool stuff) from an html document☆2,150Updated last year
- Google Search SERP Scraper☆104Updated last year
- Puppeteer (Headless Chrome Node API)-based rendering solution.☆527Updated 2 years ago
- House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.☆117Updated last year
- A collection of awesome web scaper, crawler.☆252Updated 7 months ago
- Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.☆69Updated 3 years ago
- Cloud crawler functions for scrapeulous☆44Updated 3 years ago
- Puppeteer(Chrome headless node API) based web page renderer☆314Updated last month
- Flexible event driven crawler for node.☆2,140Updated 3 years ago
- A JavaScript library for generating random user agents with data that's updated daily.☆982Updated this week
- Proxies Puppeteer Page requests.☆201Updated 2 months ago
- Small tool to wait that all xhr are finished in puppeteer☆276Updated last week
- simple multi-level scraper json input/output for Cheerio☆198Updated last year
- Instagram automation driven by headless chrome.☆115Updated last year
- Simple, lightweight and expressive web scraping with Node.js☆154Updated 3 years ago
- Chromium / Puppeteer site crawler☆48Updated 4 years ago
- Starter Kit for running Headless-Chrome by Puppeteer on AWS Lambda.☆581Updated 4 years ago
- Node module that summarizes text using a naive summarization algorithm☆769Updated 3 weeks ago
- High-performance FlexSearch Server for Node.js (Cluster)☆188Updated 5 years ago
- Plugin for website-scraper which returns html for dynamic websites using puppeteer☆324Updated last month
- Puppeteer as a service☆456Updated 2 years ago