Alternatives and similar repositories for scrapy-wayback-machine:

Users that are interested in scrapy-wayback-machine are comparing it to the libraries listed below

• sangaline / wayback-machine-scraper
  A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
  ☆430Updated 11 months ago
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆80Updated 2 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆55Updated last year
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆182Updated 6 years ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆133Updated 4 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆110Updated 2 months ago
• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆62Updated 5 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆166Updated 3 weeks ago
• mediacloud / nyt-news-labeler
  Tag news stories based on models trained on the NYT corpus.
  ☆42Updated last year
• zytedata / zyte-smartproxy-headless-proxy
  A complimentary proxy to help to use SPM with headless browsers
  ☆109Updated last year
• scrapinghub / sample-projects
  Sample projects showcasing Scrapinghub tech
  ☆138Updated 11 months ago
• scrapinghub / scrapinghub-stack-scrapy
  Software stack with latest Scrapy and updated deps
  ☆63Updated 3 weeks ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆42Updated 5 years ago
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆118Updated 7 months ago
• Webhose / article-date-extractor
  Automatically extracts and normalizes an online article or blog post publication date
  ☆118Updated last year
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆167Updated 3 weeks ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆395Updated last month
• scrapinghub / scrapy-poet
  Page Object pattern for Scrapy
  ☆119Updated this week
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆107Updated 2 weeks ago
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆148Updated 4 months ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆121Updated 5 months ago
• scrapinghub / python-scrapinghub
  A client interface for Scrapinghub's API
  ☆203Updated last month
• scrapinghub / scrapy-autoextract
  Zyte Automatic Extraction integration for Scrapy
  ☆56Updated 2 years ago
• darpa-i2o / memex-program-index
  A list of memex-related tools and their repository URLs
  ☆147Updated 6 years ago
• scrapinghub / web-poet
  Web scraping Page Objects core library
  ☆96Updated 3 months ago
• hermit-crab / ScrapeMate
  Scraping assistant tool. Editing and maintaining CSS/XPath selectors across webpages.
  ☆101Updated 6 years ago
• ThomasAitken / Scrapy-Testmaster
  The most advanced debugging and testing tool for Scrapy
  ☆16Updated last year
• internetarchive / warctools
  Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)
  ☆153Updated 4 years ago
• AnthonySigogne / keyword-mining
  API - extract a list of keywords from a text.
  ☆18Updated 7 years ago
• public-law / open-gov-crawlers
  Parse government documents into well formed JSON
  ☆66Updated last month