internetarchive / sandcrawler
Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
☆25Updated 7 months ago
Alternatives and similar repositories for sandcrawler:
Users that are interested in sandcrawler are comparing it to the libraries listed below
- Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives☆14Updated 3 years ago
- Command line tool for digging into WARC files☆38Updated last week
- Digital Preservation of HTTP in documentary heritage.☆22Updated last year
- Open Access PDF harvester☆39Updated 10 months ago
- Trough: Big data, small databases.☆40Updated 7 months ago
- CDXJ Indexing of WARC/ARCs☆25Updated 3 months ago
- Web archive index server based on RocksDB☆34Updated 4 months ago
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆44Updated 7 years ago
- Comparing warc files☆17Updated 6 years ago
- Specification for authentication and creating signed WACZ Files☆10Updated 3 years ago
- A Memento Aggregator CLI and Server in Go☆62Updated 2 weeks ago
- Make MP3 albums out of Academic PDFs. Works by gluing together Grobid and TTS offerings.☆12Updated last year
- Markdown for Linked Data☆16Updated 9 years ago
- Web application for distributed compute analysis of Archive-It web archive collections.☆16Updated this week
- search interface for scholarly works☆84Updated 7 months ago
- Sort-friendly URI Reordering Transform (SURT) python module☆41Updated 7 months ago
- dataset is a command line tool, Go package, shared library and Python package for working with JSON objects as collections☆24Updated 4 months ago
- A PDF classifier ensemble with REST API service☆23Updated 4 years ago
- ☆10Updated 3 years ago
- A tool to extract canonical references from text.☆20Updated 3 years ago
- A command line utility for listing and searching snapshots in web archives☆16Updated last year
- Support for writing WARC files with Scrapy☆21Updated 5 years ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆51Updated 6 years ago
- oldweb.today Remote/Containerized Browser System☆10Updated 6 years ago
- Upload SQLite database files to Datasette☆12Updated 10 months ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆39Updated last week
- A tool for detecting viruses and NSFW material in WARC files☆11Updated 7 months ago
- Perpetual Access To The Scholarly Record☆118Updated 7 months ago
- Downloads and imports Wikipedia page histories to a git repository☆34Updated 3 months ago