Alternatives and similar repositories for WarcMiddleware

Users that are interested in WarcMiddleware are comparing it to the libraries listed below

• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆61Updated 6 years ago
• ikreymer / pywb-webrecorder
  Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io
  ☆38Updated 10 years ago
• alard / warc-proxy
  Serving content from a WARC
  ☆62Updated 13 years ago
• pudo-attic / scrapekit
  Python library with common functionality for writing web scrapers
  ☆102Updated 10 years ago
• Alir3z4 / python-sanitize
  Bringing sanity to world of messed-up data
  ☆66Updated 11 years ago
• EugenePig / ebook-isbn
  An eBook tool to extract ISBN or Metadata form eBook and rename them by using ISBN database and Metadata
  ☆29Updated 10 years ago
• scrapy-plugins / scrapy-pagestorage
  A scrapy extension to store requests and responses information in storage service
  ☆27Updated 3 years ago
• scrapy-plugins / scrapy-splitvariants
  Scrapy spider middleware to split an item into multiple items using a multi-valued key
  ☆21Updated 8 years ago
• alard / megawarc
  Nondestructive warc-in-tar to warc conversion
  ☆27Updated 12 years ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆64Updated 5 months ago
• fullscale / pypes
  A component based data flow framework with a drag-n-drop Web 2.0 interface. Based on Stackless Python and inspired by Yahoo! Pipes.
  ☆150Updated 13 years ago
• internetarchive / warc
  Python library for reading and writing warc files
  ☆247Updated 3 years ago
• ArchiveTeam / VideoBot
  Specialised bot for periodical grabs and video/audio/etc. webpage scrapes.
  ☆11Updated 8 years ago
• toxinu / leselys
  I'm Leselys, your very elegant RSS reader.
  ☆226Updated 5 years ago
• cantabular / scraperwiki-python
  ScraperWiki Python library for scraping and saving data; in maintenance mode
  ☆158Updated last week
• Miserlou / easy-expressions
  Easy Regular Expressions for Python
  ☆24Updated 3 years ago
• redapple / parslepy
  Python implementation of the Parsley language for extracting structured data from web pages
  ☆92Updated 8 years ago
• SerenitySoftware / cahoots
  A Text Comprehension Engine in Python
  ☆15Updated 10 years ago
• iandennismiller / offline-pages
  Save a bunch of web pages as a self-contained, compressed archive file for offline storage and sharing.
  ☆35Updated 13 years ago
• peterk / warcworker
  A dockerized, queued high fidelity web archiver based on Squidwarc
  ☆61Updated last year
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated 3 weeks ago
• hatnote / cronfed
  Convert cron emails to RSS 2.0. It's the least you can do.
  ☆14Updated 10 years ago
• bdoms / tumblr_backup
  A Python script for saving your Tumblr blog to your hard drive as HTML or CSV.
  ☆88Updated 8 years ago
• webrecorder / har2warc
  Convert HTTP Archive (HAR) -> Web Archive (WARC) format
  ☆54Updated 7 years ago
• hupili / snsapi
  Cross platform middleware for Social Networking Services: Twitter, Facebook, SinaWeibo, Renren, RSS, Email, Sqlite, ... (more coming)
  ☆157Updated 3 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated this week
• scrapinghub / page_finder
  Find which links on a web page are pagination links
  ☆29Updated 9 years ago
• jabbalaci / Jabba-Webkit
  Jabba's headless webkit browser for scraping AJAX-powered webpages.
  ☆90Updated 11 years ago
• PromyLOPh / crocoite
  Web archiving using Google Chrome
  ☆46Updated 6 years ago
• cosmiclattes / wikireplay
  A javascript tool to visualize the diff's in wikipedia
  ☆36Updated 3 years ago