Alternatives and similar repositories for urlcanon

Users that are interested in urlcanon are comparing it to the libraries listed below

• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆44Updated last month
• internetarchive / warc
  Python library for reading and writing warc files
  ☆243Updated 3 years ago
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆117Updated 2 weeks ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• ukwa / webarchive-discovery
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆130Updated 3 months ago
• internetarchive / trough
  Trough: Big data, small databases.
  ☆40Updated last year
• pudo / normality
  A tiny library for Python text normalisation. Useful for ad-hoc text processing.
  ☆155Updated last month
• harvard-lil / waczerciser
  Create and edit WARC and WACZ files
  ☆17Updated 10 months ago
• edsu / microdata
  python library for extracting html microdata
  ☆166Updated 2 years ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆434Updated 10 months ago
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 3 years ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• DocNow / diffengine
  track changes to the news, where news is anything with an RSS feed
  ☆179Updated 5 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 7 years ago
• nla / outbackcdx
  Web archive index server based on RocksDB
  ☆36Updated last month
• ukwa / ukwa-pywb
  ☆11Updated 2 years ago
• mloesch / sickle
  Sickle: OAI-PMH for Humans
  ☆114Updated 2 years ago
• pykler / yajl-py
  Pure Python wrapper to the Yajl C Library
  ☆84Updated 10 months ago
• TeamHG-Memex / soft404
  A classifier for detecting soft 404 pages
  ☆56Updated 2 weeks ago
• internetarchive / warctools
  Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)
  ☆166Updated 2 months ago
• pediapress / mwlib
  mediawiki parser library
  ☆105Updated 2 months ago
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆29Updated 10 months ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆134Updated 5 years ago
• dhamaniasad / WARCTools
  A list of tools related to W(eb)ARC(hive)
  ☆64Updated 10 years ago
• DocNow / waybackprov
  utility to fetch provenance information from Internet Archive's Wayback Machine
  ☆13Updated 3 years ago
• legoktm / toolforge
  [OBSOLETE] Replaced by https://gitlab.wikimedia.org/toolforge-repos/python-toolforge
  ☆22Updated 2 years ago
• unt-libraries / py-wasapi-client
  A client for the Archive-It And Webrecorder WASAPI Data Transfer API
  ☆16Updated 6 years ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆63Updated 2 months ago
• niksite / url-normalize
  URL normalization for Python
  ☆99Updated 6 months ago
• seomoz / url-py
  URL Transformation, Sanitization
  ☆103Updated last year