Alternatives and similar repositories for Crawling-Infrastructure

Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below

• NikolaiT / se-scraper
  Javascript scraping module based on puppeteer for many different search engines...
  ☆559Updated 2 years ago
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆45Updated 4 years ago
• NikolaiT / struktur
  Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.
  ☆70Updated 3 years ago
• paulirish / headless-cat-n-mouse
  Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.
  ☆657Updated 3 years ago
• NikolaiT / stealthy-scraping-tools
  Minimal set of tools to conduct stealthy scraping.
  ☆156Updated 2 years ago
• linkedtales / scrapedin
  LinkedIn Scraper (currently working 2020)
  ☆607Updated 2 years ago
• unblocked-web / double-agent
  A test suite of common scraper detection techniques. See how detectable your scraper stack is.
  ☆136Updated 2 years ago
• apify / browser-pool
  A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppetee…
  ☆94Updated 2 years ago
• antoinevastel / bots-zoo
  ☆115Updated last year
• linkedtales / scrapedin-linkedin-crawler
  Crawler for LinkedIn full profiles 2019
  ☆215Updated 4 years ago
• TeamHG-Memex / scrapy-rotating-proxies
  use multiple proxies with Scrapy
  ☆760Updated 3 years ago
• TheWebScrapingClub / TheScrapingClubFree
  The Web Scraping Club Free Repository
  ☆144Updated 3 weeks ago
• ulixee / secret-agent
  The web scraper that's nearly impossible to block - now called @ulixee/hero
  ☆712Updated 2 years ago
• chris124567 / puppeteer-bypassing-bot-detection
  Bypassing bot detection checks with Puppeteer.
  ☆93Updated 4 years ago
• transitive-bullshit / puppeteer-email
  Email automation driven by headless chrome.
  ☆167Updated 4 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆119Updated 2 weeks ago
• danielgatis / puppeteer-recaptcha-solver
  Recaptcha solver for puppeteer.
  ☆609Updated 2 months ago
• apify / proxy-chain
  Node.js implementation of a proxy server (think Squid) with support for SSL, authentication and upstream proxy chaining.
  ☆915Updated last month
• antoinevastel / fpscanner
  ☆574Updated 3 months ago
• zlurad / serp-parser
  Nodejs lib to parse Google SERP html pages
  ☆47Updated last year
• apify / actor-scraper
  House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
  ☆121Updated 2 years ago
• prescience-data / foundation
  🧱 A uniform template to use as a foundation for Puppeteer bot construction.
  ☆66Updated 4 years ago
• gajus / puppeteer-proxy
  Proxies Puppeteer Page requests.
  ☆208Updated 9 months ago
• brendonboshell / supercrawler
  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…
  ☆381Updated 2 years ago
• intoli / user-agents
  A JavaScript library for generating random user agents with data that's updated daily.
  ☆1,054Updated this week
• dandrews / nefarious-linkedin
  A look at how LinkedIn spies on its users.
  ☆829Updated 6 years ago
• bitmakerla / estela
  estela, an elastic web scraping cluster 🕸
  ☆181Updated this week
• scrapinghub / spidermon
  Scrapy Extension for monitoring spiders execution.
  ☆539Updated last month
• antoinevastel / fp-collect
  Fingerprinting script of Fingerprint-Scanner
  ☆248Updated 3 months ago
• cassidoo / scrapers
  A list of scrapers from around the web.
  ☆673Updated 3 months ago