Alternatives and similar repositories for urlcanon

Users that are interested in urlcanon are comparing it to the libraries listed below

• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆43Updated 3 weeks ago
• internetarchive / warc
  Python library for reading and writing warc files
  ☆244Updated 3 years ago
• ukwa / webarchive-discovery
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆128Updated 2 months ago
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆117Updated 10 months ago
• harvard-lil / waczerciser
  Create and edit WARC and WACZ files
  ☆17Updated 9 months ago
• internetarchive / warctools
  Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)
  ☆165Updated last month
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• internetarchive / trough
  Trough: Big data, small databases.
  ☆41Updated last year
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 3 years ago
• nla / outbackcdx
  Web archive index server based on RocksDB
  ☆35Updated last month
• scrapy / protego
  A pure-Python robots.txt parser with support for modern conventions.
  ☆70Updated 2 months ago
• pykler / yajl-py
  Pure Python wrapper to the Yajl C Library
  ☆84Updated 9 months ago
• sethmlarson / whatwg-url
  Python implementation of WHATWG URL Living Standard
  ☆21Updated last year
• iipc / openwayback
  The OpenWayback Development
  ☆504Updated last year
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆62Updated last month
• nlnwa / warchaeology
  Command line tool for digging into WARC files
  ☆46Updated this week
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆28Updated 9 months ago
• unt-libraries / py-wasapi-client
  A client for the Archive-It And Webrecorder WASAPI Data Transfer API
  ☆16Updated 5 years ago
• webrecorder / warcit
  Convert Directories, Files and ZIP Files to Web Archives (WARC)
  ☆87Updated 5 months ago
• mementoweb / py-memento-client
  A Memento Client Library in Python
  ☆26Updated 7 years ago
• WASAPI-Community / data-transfer-apis
  WASAPI data transfer APIs
  ☆47Updated 3 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆131Updated last week
• niksite / url-normalize
  URL normalization for Python
  ☆98Updated 5 months ago
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆152Updated 2 months ago
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 2 years ago
• iipc / jwarc
  Java library for reading and writing WARC files with a typed API
  ☆50Updated 2 weeks ago
• edsu / microdata
  python library for extracting html microdata
  ☆167Updated 2 years ago
• TeamHG-Memex / undercrawler
  A generic crawler
  ☆78Updated 7 years ago
• chfoo / warcat
  Tool and library for handling Web ARChive (WARC) files.
  ☆164Updated 11 months ago
• mloesch / sickle
  Sickle: OAI-PMH for Humans
  ☆114Updated 2 years ago