odie5533 / WarcMiddlewareLinks
WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.
☆47Updated 7 years ago
Alternatives and similar repositories for WarcMiddleware
Users that are interested in WarcMiddleware are comparing it to the libraries listed below
Sorting:
- Serving content from a WARC☆62Updated 12 years ago
- Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io☆38Updated 9 years ago
- HTTP(S) proxy that saves traffic to a WARC file, using libmitmproxy.☆16Updated 11 years ago
- Bringing sanity to world of messed-up data☆66Updated 10 years ago
- An eBook tool to extract ISBN or Metadata form eBook and rename them by using ISBN database and Metadata☆30Updated 10 years ago
- Jabba's headless webkit browser for scraping AJAX-powered webpages.☆91Updated 10 years ago
- Python implementation of the Parsley language for extracting structured data from web pages☆92Updated 7 years ago
- Grabbing all news.☆62Updated 5 years ago
- I'm Leselys, your very elegant RSS reader.☆226Updated 4 years ago
- A queue-controlled browser automation tool for improving web crawl quality☆61Updated 3 months ago
- Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations☆40Updated last year
- Primary LocalWiki backend server environment☆48Updated 7 years ago
- Python library with common functionality for writing web scrapers☆102Updated 10 years ago
- oldweb.today Remote/Containerized Browser System☆10Updated 6 years ago
- Scrapy middleware for the autologin☆37Updated 7 years ago
- Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application☆24Updated 4 years ago
- Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation fo…☆102Updated last year
- A clean-room clone of the Fever RSS aggregator, focusing on the API☆61Updated 3 years ago
- A component that tries to avoid downloading duplicate content☆27Updated 7 years ago
- Easy Regular Expressions for Python☆24Updated 2 years ago
- IRC web client and bot framework☆160Updated 9 years ago
- PyQuery-based scraping micro-framework.☆117Updated 3 years ago
- Scrapy middleware which allows to crawl only new content☆80Updated 2 years ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆52Updated 6 years ago
- A small python script for easy access to firefox bookmarks and browsing history☆22Updated 5 years ago
- Define simple search patterns in bulk to perform advanced matching on any string☆56Updated last year
- Search in multiple torrent sites from your CLI☆69Updated 2 years ago
- A dockerized, queued high fidelity web archiver based on Squidwarc☆61Updated last year
- ScraperWiki Python library for scraping and saving data☆159Updated 2 years ago
- Trough: Big data, small databases.☆42Updated 11 months ago