Alternatives and similar repositories for Crawling-Infrastructure:

Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below

• NikolaiT / se-scraper
  Javascript scraping module based on puppeteer for many different search engines...
  ☆553Updated 2 years ago
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆45Updated 3 years ago
• unblocked-web / double-agent
  A test suite of common scraper detection techniques. See how detectable your scraper stack is.
  ☆136Updated 2 years ago
• NikolaiT / struktur
  Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.
  ☆69Updated 3 years ago
• NikolaiT / stealthy-scraping-tools
  Minimal set of tools to conduct stealthy scraping.
  ☆154Updated last year
• TeamHG-Memex / scrapy-rotating-proxies
  use multiple proxies with Scrapy
  ☆751Updated 2 years ago
• paulirish / headless-cat-n-mouse
  Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.
  ☆649Updated 3 years ago
• prescience-data / foundation
  🧱 A uniform template to use as a foundation for Puppeteer bot construction.
  ☆65Updated 3 years ago
• brendonboshell / supercrawler
  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…
  ☆379Updated 2 years ago
• ulixee / secret-agent
  The web scraper that's nearly impossible to block - now called @ulixee/hero
  ☆701Updated last year
• antoinevastel / bots-zoo
  ☆111Updated 11 months ago
• gajus / puppeteer-proxy
  Proxies Puppeteer Page requests.
  ☆207Updated 5 months ago
• linkedtales / scrapedin
  LinkedIn Scraper (currently working 2020)
  ☆598Updated last year
• freethenation / DFPM
  DFPM is a browser extension for detecting browser fingerprinting.
  ☆116Updated 2 years ago
• TheWebScrapingClub / TheScrapingClubFree
  The Web Scraping Club Free Repository
  ☆136Updated 3 months ago
• prescience-data / puppeteer-botcheck
  🕵‍♂ Bot detection tests for Puppeteer. Hide and seek!
  ☆87Updated last year
• antoinevastel / fp-collect
  Fingerprinting script of Fingerprint-Scanner
  ☆244Updated 11 months ago
• antoinevastel / fpscanner
  ☆554Updated 11 months ago
• clemfromspace / scrapy-cloudflare-middleware
  A Scrapy middleware to bypass the CloudFlare's anti-bot protection
  ☆106Updated 3 years ago
• Cuadrix / puppeteer-page-proxy
  Additional module to use with 'puppeteer' for setting proxies per page basis.
  ☆439Updated 8 months ago
• apify / actor-scraper
  House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
  ☆119Updated last year
• apify / browser-pool
  A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppetee…
  ☆94Updated 2 years ago
• apify / header-generator
  NodeJs package for generating browser-like headers.
  ☆66Updated 2 years ago
• luminati-io / luminati-proxy
  Luminati HTTP/HTTPS Proxy manager
  ☆754Updated 2 weeks ago
• apify / proxy-chain
  Node.js implementation of a proxy server (think Squid) with support for SSL, authentication and upstream proxy chaining.
  ☆890Updated 3 weeks ago
• linkedtales / scrapedin-linkedin-crawler
  Crawler for LinkedIn full profiles 2019
  ☆215Updated 4 years ago
• chris124567 / puppeteer-bypassing-bot-detection
  Bypassing bot detection checks with Puppeteer.
  ☆93Updated 4 years ago
• intoli / user-agents
  A JavaScript library for generating random user agents with data that's updated daily.
  ☆1,017Updated this week
• danielgatis / puppeteer-recaptcha-solver
  Recaptcha solver for puppeteer.
  ☆596Updated 9 months ago
• apify / fingerprint-suite
  Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
  ☆1,189Updated this week