Alternatives and similar repositories for Crawling-Infrastructure:

Users that are interested in Crawling-Infrastructure are comparing it to the libraries listed below

• NikolaiT / se-scraper
  Javascript scraping module based on puppeteer for many different search engines...
  ☆556Updated 2 years ago
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆45Updated 4 years ago
• NikolaiT / struktur
  Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.
  ☆69Updated 3 years ago
• NikolaiT / stealthy-scraping-tools
  Minimal set of tools to conduct stealthy scraping.
  ☆155Updated last year
• unblocked-web / double-agent
  A test suite of common scraper detection techniques. See how detectable your scraper stack is.
  ☆136Updated 2 years ago
• cassidoo / scrapers
  A list of scrapers from around the web.
  ☆659Updated last month
• TeamHG-Memex / scrapy-rotating-proxies
  use multiple proxies with Scrapy
  ☆754Updated 2 years ago
• ulixee / secret-agent
  The web scraper that's nearly impossible to block - now called @ulixee/hero
  ☆705Updated 2 years ago
• paulirish / headless-cat-n-mouse
  Is headless chrome currently detectable? Let's pit the detections and detection evasions against eachother.
  ☆654Updated 3 years ago
• apify / browser-pool
  A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppetee…
  ☆94Updated 2 years ago
• brendonboshell / supercrawler
  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and con…
  ☆380Updated 2 years ago
• apify / actor-scraper
  House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
  ☆120Updated last year
• prescience-data / foundation
  🧱 A uniform template to use as a foundation for Puppeteer bot construction.
  ☆65Updated 3 years ago
• Niek / playwright-addons
  Add-ons for Playwright: adblocker, stealth mode
  ☆46Updated 4 years ago
• linkedtales / scrapedin
  LinkedIn Scraper (currently working 2020)
  ☆601Updated 2 years ago
• christophebe / serp
  Google Search SERP Scraper
  ☆107Updated last year
• currentslab / extractnet
  A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one pac…
  ☆273Updated last year
• apify / got-scraping
  HTTP client made for scraping based on got.
  ☆637Updated last month
• 39ff / docker-rotating-proxy
  create your rotating proxy server with docker. self hosted rotating proxy service.
  ☆174Updated 2 years ago
• freethenation / DFPM
  DFPM is a browser extension for detecting browser fingerprinting.
  ☆116Updated 2 years ago
• zlurad / serp-parser
  Nodejs lib to parse Google SERP html pages
  ☆46Updated last year
• TheWebScrapingClub / TheScrapingClubFree
  The Web Scraping Club Free Repository
  ☆137Updated 4 months ago
• microlinkhq / browserless
  The headless Chrome/Chromium driver on top of Puppeteer.
  ☆1,692Updated last week
• scrapinghub / extruct
  Extract embedded metadata from HTML markup
  ☆897Updated last month
• apify / proxy-chain
  Node.js implementation of a proxy server (think Squid) with support for SSL, authentication and upstream proxy chaining.
  ☆898Updated this week
• linkedtales / scrapedin-linkedin-crawler
  Crawler for LinkedIn full profiles 2019
  ☆215Updated 4 years ago
• chris124567 / puppeteer-bypassing-bot-detection
  Bypassing bot detection checks with Puppeteer.
  ☆93Updated 4 years ago
• antoinevastel / bots-zoo
  ☆113Updated last year
• ecoron / SerpScrap
  SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type …
  ☆261Updated 2 years ago
• sangaline / scrapy-wayback-machine
  A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
  ☆113Updated last year