odie5533 / WarcMiddlewareLinks
WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.
☆47Updated 7 years ago
Alternatives and similar repositories for WarcMiddleware
Users that are interested in WarcMiddleware are comparing it to the libraries listed below
Sorting:
- Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io☆38Updated 10 years ago
- Serving content from a WARC☆62Updated 12 years ago
- Bringing sanity to world of messed-up data☆66Updated 11 years ago
- Grabbing all news.☆62Updated 6 years ago
- A queue-controlled browser automation tool for improving web crawl quality☆63Updated 4 months ago
- A component based data flow framework with a drag-n-drop Web 2.0 interface. Based on Stackless Python and inspired by Yahoo! Pipes.☆150Updated 13 years ago
- A dockerized, queued high fidelity web archiver based on Squidwarc☆61Updated last year
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆54Updated 7 years ago
- Nondestructive warc-in-tar to warc conversion☆27Updated 12 years ago
- An eBook tool to extract ISBN or Metadata form eBook and rename them by using ISBN database and Metadata☆29Updated 10 years ago
- "Old SFM" -- manage rules and streams from social data sources, starting with twitter.☆86Updated 2 years ago
- I'm Leselys, your very elegant RSS reader.☆226Updated 5 years ago
- Trough: Big data, small databases.☆41Updated last year
- Scrapy middleware which allows to crawl only new content☆79Updated this week
- (Note: This repository is obsolete, please see the new Browsertrix webrecorder/browsertrix) Browser-Based On-Demand Web Archiving Automat…☆38Updated 6 years ago
- Specialised bot for periodical grabs and video/audio/etc. webpage scrapes.☆11Updated 8 years ago
- Easy Regular Expressions for Python☆24Updated 2 years ago
- Python implementation of the Parsley language for extracting structured data from web pages☆92Updated 8 years ago
- Python scripts for scraping bus ticket data from the websites of BoltBus, Greyhound, Megabus, GoBus, Amtrak, Peterpan, and EasternTravel.☆38Updated 5 years ago
- Web archiving using Google Chrome☆46Updated 5 years ago
- Python library for reading and writing warc files☆246Updated 3 years ago
- A javascript tool to visualize the diff's in wikipedia☆35Updated 3 years ago
- Python library with common functionality for writing web scrapers☆102Updated 10 years ago
- A small python script for easy access to firefox bookmarks and browsing history☆23Updated 5 years ago
- Save a bunch of web pages as a self-contained, compressed archive file for offline storage and sharing.☆35Updated 13 years ago
- Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application☆24Updated 5 years ago
- NOTE: This project is no longer being actively developed.. Check out https://replayweb.page / https://github.com/webrecorder/replayweb.pa…☆200Updated 11 months ago
- IRC web client and bot framework☆160Updated 10 years ago
- Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations☆40Updated last year
- PyQuery-based scraping micro-framework.☆119Updated 3 years ago