Alternatives and similar repositories for Crawling-Infrastructure:

Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below

• NikolaiT / se-scraper
  Javascript scraping module based on puppeteer for many different search engines...
  ☆550Updated last year
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆44Updated 3 years ago
• NikolaiT / stealthy-scraping-tools
  Minimal set of tools to conduct stealthy scraping.
  ☆150Updated last year
• NikolaiT / struktur
  Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.
  ☆69Updated 3 years ago
• apify / got-scraping
  HTTP client made for scraping based on got.
  ☆560Updated 3 weeks ago
• paulirish / headless-cat-n-mouse
  Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.
  ☆646Updated 3 years ago
• unblocked-web / double-agent
  A test suite of common scraper detection techniques. See how detectable your scraper stack is.
  ☆136Updated 2 years ago
• brendonboshell / supercrawler
  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…
  ☆380Updated last year
• freethenation / DFPM
  DFPM is a browser extension for detecting browser fingerprinting.
  ☆114Updated 2 years ago
• prescience-data / foundation
  🧱 A uniform template to use as a foundation for Puppeteer bot construction.
  ☆65Updated 3 years ago
• christophebe / serp
  Google Search SERP Scraper
  ☆105Updated last year
• apify / actor-scraper
  House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
  ☆119Updated last year
• ulixee / secret-agent
  The web scraper that's nearly impossible to block - now called @ulixee/hero
  ☆676Updated last year
• linkedtales / scrapedin
  LinkedIn Scraper (currently working 2020)
  ☆597Updated last year
• TheWebScrapingClub / TheScrapingClubFree
  The Web Scraping Club Free Repository
  ☆129Updated last month
• linkedtales / scrapedin-linkedin-crawler
  Crawler for LinkedIn full profiles 2019
  ☆215Updated 4 years ago
• apify / header-generator
  NodeJs package for generating browser-like headers.
  ☆63Updated 2 years ago
• antoinevastel / fpscanner
  ☆542Updated 9 months ago
• ecoron / SerpScrap
  SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type …
  ☆259Updated 2 years ago
• apify / fingerprint-suite
  Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
  ☆1,078Updated this week
• scrapoxy / scrapoxy
  Scrapoxy is a super proxies manager that orchestrates all your proxies into one place, rather than spreading management across multiple s…
  ☆2,085Updated this week
• apify / browser-pool
  A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppetee…
  ☆91Updated 2 years ago
• TeamHG-Memex / scrapy-rotating-proxies
  use multiple proxies with Scrapy
  ☆740Updated 2 years ago
• clemfromspace / scrapy-cloudflare-middleware
  A Scrapy middleware to bypass the CloudFlare's anti-bot protection
  ☆106Updated 3 years ago
• apify / proxy-chain
  Node.js implementation of a proxy server (think Squid) with support for SSL, authentication and upstream proxy chaining.
  ☆856Updated 2 weeks ago
• zytedata / zyte-smartproxy-headless-proxy
  A complimentary proxy to help to use SPM with headless browsers
  ☆110Updated last year
• luminati-io / luminati-proxy
  Luminati HTTP/HTTPS Proxy manager
  ☆749Updated 3 weeks ago
• bitmakerla / estela
  estela, an elastic web scraping cluster 🕸
  ☆174Updated last month
• scrapinghub / spidermon
  Scrapy Extension for monitoring spiders execution.
  ☆534Updated last week