iipc / urlcanon
url canonicalization library for python and java
☆34Updated 2 years ago
Alternatives and similar repositories for urlcanon:
Users that are interested in urlcanon are comparing it to the libraries listed below
- Sort-friendly URI Reordering Transform (SURT) python module☆42Updated 8 months ago
- Webrecorders DevTools Protocol Automation Library☆17Updated 2 years ago
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆45Updated 7 years ago
- Trough: Big data, small databases.☆41Updated 8 months ago
- CDXJ Indexing of WARC/ARCs☆25Updated 4 months ago
- Delayed iteration for polling and retries.☆30Updated 4 months ago
- Parsing and validation of URIs (RFC 3896) and IRIs (RFC 3987)☆46Updated last year
- Commons of stupid, simple Python micro functions. Pull requests very welcome.☆19Updated this week
- Microformats2 parser written in Python☆106Updated 5 months ago
- Extends zip() and itertools.zip_longest() to generate named tuples.☆22Updated 5 years ago
- Utility library to turn country names into ISO two-letter codes☆66Updated last month
- Tools for helping you work with web platform archive downloads.☆17Updated 5 years ago
- A component that tries to avoid downloading duplicate content☆27Updated 6 years ago
- A pure-Python robots.txt parser with support for modern conventions.☆65Updated 3 weeks ago
- Code metrics for Python code.☆10Updated 10 years ago
- Web archiving using Google Chrome☆44Updated 5 years ago
- Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application☆24Updated 4 years ago
- Convert HTTP Archive (HAR) -> Web Archive (WARC) format☆51Updated 6 years ago
- Centralised repository for WARC usage specifications.☆109Updated 4 months ago
- Sickle: OAI-PMH for Humans☆110Updated last year
- A fork of http://pydispatcher.sourceforge.net/ with PyPy support☆16Updated 7 years ago
- URL normalization for Python☆94Updated last week
- Links recognition library with full unicode support☆18Updated last month
- A helper library full of URL-related heuristics.☆69Updated 3 weeks ago
- Python implementation of WHATWG URL Living Standard☆21Updated 9 months ago
- Python library for reading and writing warc files☆239Updated 3 years ago
- Asynchronous Server Side Events (SSE) client for Python 3☆25Updated last month
- automatic persistence for Python objects☆46Updated last month
- A Memento Client Library in Python☆26Updated 7 years ago
- Github mirror of "analytics/quarry/web" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_acce…☆43Updated 2 years ago