Alternatives and similar repositories for scrapy-wayback-machine:

Users that are interested in scrapy-wayback-machine are comparing it to the libraries listed below

• sangaline / wayback-machine-scraper
  A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
  ☆432Updated last year
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆188Updated 6 years ago
• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆62Updated 5 years ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆134Updated 4 years ago
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆118Updated 8 months ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆112Updated 3 months ago
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆80Updated 2 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆167Updated 2 months ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆56Updated last year
• scrapinghub / autoextract-spiders
  Pre-built Scrapy spiders for AutoExtract
  ☆19Updated 10 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 2 years ago
• opensanctions / fingerprints
  Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
  ☆148Updated last month
• N0taN3rd / Squidwarc
  Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
  ☆171Updated 4 years ago
• scrapinghub / scrapinghub-stack-scrapy
  Software stack with latest Scrapy and updated deps
  ☆63Updated 2 weeks ago
• oduwsdl / CarbonDate
  Estimating the age of web resources
  ☆95Updated last year
• cldellow / real-estate-prices-cc
  Source real estate prices from the Common Crawl.
  ☆27Updated 6 years ago
• TeamHG-Memex / autopager
  Detect and classify pagination links
  ☆102Updated 4 years ago
• scrapinghub / web-poet
  Web scraping Page Objects core library
  ☆96Updated 3 weeks ago
• TeamHG-Memex / aquarium
  Splash + HAProxy + Docker Compose
  ☆197Updated 6 years ago
• minimaxir / facebook-ad-library-scraper
  A Python scraper for the Facebook Ad Library, using the official Facebook Ad Library API.
  ☆118Updated 5 years ago
• GateNLP / ultimate-sitemap-parser
  Ultimate Website Sitemap Parser
  ☆193Updated last week
• scrapinghub / scrapy-poet
  Page Object pattern for Scrapy
  ☆120Updated 2 weeks ago
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆193Updated 7 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• brandonrobertz / autoscrape-py
  An automated, programming-free web scraper for interactive sites
  ☆109Updated last year
• ecoron / SerpScrap
  SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type …
  ☆260Updated 2 years ago
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆45Updated 4 years ago
• tpeng / googlesearch
  Scrape the Google search result with Scrapy.
  ☆98Updated 5 years ago
• zytedata / zyte-autoextract
  Python clients for Zyte AutoExtract API
  ☆40Updated 3 years ago