internetarchive / scrapy-warcio
Support for writing WARC files with Scrapy
☆19Updated 4 years ago
Related projects: ⓘ
- CDXJ Indexing of WARC/ARCs☆21Updated 3 months ago
- Centralised repository for WARC usage specifications.☆98Updated last month
- Web archive index server based on RocksDB☆31Updated last week
- Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki☆24Updated last month
- Command line tool for digging into WARC files☆31Updated 2 weeks ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆44Updated 5 years ago
- Chrome extension that uses Memento to indicate that a page a user is viewing on the live web has an archived copy and to give the user ac…☆49Updated 2 months ago
- Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives☆13Updated 3 years ago
- Converts WARC files to static HTML☆38Updated 2 months ago
- A Memento Aggregator CLI and Server in Go☆55Updated 4 months ago
- Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io☆39Updated 8 years ago
- ☆36Updated 5 months ago
- Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)☆149Updated 4 years ago
- Nondestructive warc-in-tar to warc conversion☆25Updated 11 years ago
- A PDF classifier ensemble with REST API service☆23Updated 3 years ago
- Command line tool to convert a file in the WARC format to a file in the ZIM format☆44Updated last week
- Tool and library for handling Web ARChive (WARC) files.☆147Updated last year
- Digital Preservation of HTTP in documentary heritage.☆22Updated last year
- A collection of tools for archiving and analysing the internet.☆68Updated 2 years ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆33Updated last month
- Sort-friendly URI Reordering Transform (SURT) python module☆39Updated last month
- Specifications developed and maintained by the Webrecorder community.☆122Updated last month
- A list of things related to software, literature, and other content for 🕣 Memento☆85Updated 3 months ago
- Comparing warc files☆14Updated 5 years ago
- Convert Directories, Files and ZIP Files to Web Archives (WARC)☆81Updated last month
- CLI implementation of httpreserve that can test links and retrieve internet archive replacements☆9Updated last year
- A social media open post web archiving tool☆25Updated 3 months ago
- Trough: Big data, small databases.☆38Updated last month
- A command line utility for listing and searching snapshots in web archives☆13Updated 9 months ago
- A command line tool to archive a git repository from GitHub to the Internet Archive.☆89Updated 3 years ago