iipc / urlcanon
url canonicalization library for python and java
☆34Updated 2 years ago
Alternatives and similar repositories for urlcanon:
Users that are interested in urlcanon are comparing it to the libraries listed below
- Sort-friendly URI Reordering Transform (SURT) python module☆41Updated 7 months ago
- Trough: Big data, small databases.☆40Updated 7 months ago
- CDXJ Indexing of WARC/ARCs☆25Updated 3 months ago
- Webrecorders DevTools Protocol Automation Library☆17Updated 2 years ago
- A queue-controlled browser automation tool for improving web crawl quality☆60Updated last week
- ☆11Updated last year
- Nondestructive warc-in-tar to warc conversion☆26Updated 11 years ago
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆44Updated 7 years ago
- A client for the Archive-It And Webrecorder WASAPI Data Transfer API☆16Updated 5 years ago
- Centralised repository for WARC usage specifications.☆108Updated 4 months ago
- A Memento Aggregator CLI and Server in Go☆62Updated 2 weeks ago
- Grabbing all news.☆62Updated 5 years ago
- Web application for distributed compute analysis of Archive-It web archive collections.☆16Updated this week
- Serving content from a WARC☆61Updated 12 years ago
- Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)☆158Updated 4 years ago
- Command line tool for digging into WARC files☆38Updated this week
- Python library for reading and writing warc files☆239Updated 3 years ago
- Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application☆24Updated 4 years ago
- Utility library to turn country names into ISO two-letter codes☆66Updated last month
- Tools for helping you work with web platform archive downloads.☆17Updated 4 years ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆39Updated last week
- Tool and library for handling Web ARChive (WARC) files.☆156Updated 5 months ago
- wabac.js - Web Archive Browsing Augmentation Client☆106Updated this week
- The god of human readable numbers☆12Updated 5 years ago
- extract difference between two html pages☆32Updated 6 years ago
- A pure-Python robots.txt parser with support for modern conventions.☆60Updated last week
- ☆10Updated 3 years ago
- Web archive index server based on RocksDB☆34Updated 4 months ago