iipc / urlcanon
url canonicalization library for python and java
☆33Updated 2 years ago
Related projects ⓘ
Alternatives and complementary repositories for urlcanon
- Sort-friendly URI Reordering Transform (SURT) python module☆40Updated 3 months ago
- Webrecorders DevTools Protocol Automation Library☆17Updated 2 years ago
- Centralised repository for WARC usage specifications.☆100Updated 2 months ago
- CDXJ Indexing of WARC/ARCs☆21Updated this week
- Trough: Big data, small databases.☆38Updated 3 months ago
- A queue-controlled browser automation tool for improving web crawl quality☆60Updated 4 years ago
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆42Updated 6 years ago
- A component that tries to avoid downloading duplicate content☆27Updated 6 years ago
- Web archive index server based on RocksDB☆32Updated 3 weeks ago
- Tools for helping you work with web platform archive downloads.☆17Updated 4 years ago
- WARC and ARC indexing and discovery tools.☆116Updated 3 months ago
- Command line tool for digging into WARC files☆34Updated last week
- A listing of world wide web archives, for humans and machines using Web Archive Manifest (WAM) yaml format☆43Updated last year
- The god of human readable numbers☆12Updated 5 years ago
- Pluggable DSL that uses pipes to perform a series of linear transformations to extract data☆15Updated 4 months ago
- Utility library to turn country names into ISO two-letter codes☆66Updated 3 weeks ago
- Web archiving using Google Chrome☆42Updated 4 years ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆45Updated 6 years ago
- Seeder - Czech webarchive curating tool and public site☆15Updated 2 weeks ago
- Serving content from a WARC☆60Updated 11 years ago
- Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io☆39Updated 9 years ago
- Common web archive utility code.☆50Updated 3 weeks ago
- A prototype server to swarm multiple DATs for Webrecorder☆13Updated 5 years ago
- ☆11Updated last year
- utility to fetch provenance information from Internet Archive's Wayback Machine☆13Updated 2 years ago
- Java library for reading and writing WARC files with a typed API☆47Updated 3 weeks ago
- Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application☆22Updated 4 years ago
- WASAPI data transfer APIs☆42Updated 2 years ago
- A Memento Client Library in Python☆25Updated 6 years ago
- Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archive…☆22Updated last year