Alternatives and similar repositories for scrapy-wayback-machine

Users that are interested in scrapy-wayback-machine are comparing it to the libraries listed below

• sangaline / wayback-machine-scraper
  A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
  ☆452Updated last year
• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆62Updated 5 years ago
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆80Updated 2 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆178Updated 6 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• openeventdata / scraper
  Scrapes sites. Gets news. Eventually events.
  ☆87Updated 9 years ago
• minimaxir / facebook-ad-library-scraper
  A Python scraper for the Facebook Ad Library, using the official Facebook Ad Library API.
  ☆119Updated 5 years ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• mediacloud / nyt-news-labeler
  Tag news stories based on models trained on the NYT corpus.
  ☆42Updated 2 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆119Updated last year
• scrapinghub / scrapinghub-stack-scrapy
  Software stack with latest Scrapy and updated deps
  ☆63Updated this week
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆134Updated 4 years ago
• GateNLP / ultimate-sitemap-parser
  Ultimate Website Sitemap Parser
  ☆222Updated last month
• opensanctions / fingerprints
  Now included in rigour
  ☆151Updated 2 months ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆123Updated 2 months ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆127Updated 2 weeks ago
• NikolaiT / scrapeulous
  Cloud crawler functions for scrapeulous
  ☆45Updated 4 years ago
• DocNow / diffengine
  track changes to the news, where news is anything with an RSS feed
  ☆178Updated 5 years ago
• justinlittman / fbarc
  A commandline tool and Python library for archiving data from Facebook using the Graph API.
  ☆78Updated 7 years ago
• freelawproject / courts-db
  A database of courts, tests and other experiments
  ☆83Updated last week
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆195Updated 6 years ago
• public-law / open-gov-crawlers
  Parse government documents into well formed JSON
  ☆70Updated last month
• cldellow / real-estate-prices-cc
  Source real estate prices from the Common Crawl.
  ☆27Updated 6 years ago
• zytedata / zyte-smartproxy-headless-proxy
  A complimentary proxy to help to use SPM with headless browsers
  ☆108Updated 2 years ago
• brandonrobertz / autoscrape-py
  An automated, programming-free web scraper for interactive sites
  ☆111Updated 2 years ago
• johnbumgarner / newspaper3_usage_overview
  This repository provides usage examples for the Python module Newspaper3k.
  ☆147Updated last year
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆133Updated 6 months ago
• alephdata / alephclient
  API client for Aleph, supports bulk entity and document upload.
  ☆28Updated 9 months ago
• N0taN3rd / Squidwarc
  Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
  ☆170Updated 5 years ago