odie5533 / WarcMiddlewareLinks
WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.
☆46Updated 7 years ago
Alternatives and similar repositories for WarcMiddleware
Users that are interested in WarcMiddleware are comparing it to the libraries listed below
Sorting:
- Serving content from a WARC☆61Updated 12 years ago
- Bringing sanity to world of messed-up data☆66Updated 10 years ago
- A component that tries to avoid downloading duplicate content☆27Updated 7 years ago
- Grabbing all news.☆62Updated 5 years ago
- Web archiving using Google Chrome☆44Updated 5 years ago
- Scrapy spider middleware to split an item into multiple items using a multi-valued key☆20Updated 8 years ago
- Easy Regular Expressions for Python☆24Updated 2 years ago
- Tag-based bookmark manager inspired by delicious and Pinboard☆34Updated 2 years ago
- A scrapy extension to store requests and responses information in storage service☆26Updated 3 years ago
- A small python script for easy access to firefox bookmarks and browsing history☆22Updated 5 years ago
- Scrapy middleware which allows to crawl only new content☆79Updated 2 years ago
- An eBook tool to extract ISBN or Metadata form eBook and rename them by using ISBN database and Metadata☆30Updated 9 years ago
- Install python dependencies automatically at runtime☆13Updated 9 years ago
- Hash-based password manager☆19Updated 6 years ago
- video indexing site☆217Updated 9 years ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆51Updated 6 years ago
- A cookiecutter template for creating reusable Bottle projects quickly.☆28Updated 9 years ago
- PyQuery-based scraping micro-framework.☆116Updated 3 years ago
- Specialised bot for periodical grabs and video/audio/etc. webpage scrapes.☆11Updated 7 years ago
- Paginating the web☆37Updated 11 years ago
- A Python client for Chrome's DevTools protocol / a headless chrome control library☆15Updated 6 years ago
- Access domain information via python and command line.☆16Updated last year
- Jabba's headless webkit browser for scraping AJAX-powered webpages.☆92Updated 10 years ago
- A simple, system independent infrastructure for performing web scraping. Utilizes Vagrant virtualbox interface and puppet provisioning to…☆24Updated 10 years ago
- Python script for searching through your digital books and cataloguing them in an easy-to-share list of files.☆31Updated 5 years ago
- Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations☆40Updated last year
- Python library with common functionality for writing web scrapers☆102Updated 9 years ago
- Saves proxied HTTP traffic to a WARC file.☆28Updated 11 years ago
- A clean-room clone of the Fever RSS aggregator, focusing on the API☆61Updated 3 years ago
- Python implementation of the Parsley language for extracting structured data from web pages☆92Updated 7 years ago