Alternatives and similar repositories for urlcanon

Users that are interested in urlcanon are comparing it to the libraries listed below

• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆44Updated 3 months ago
• internetarchive / warc
  Python library for reading and writing warc files
  ☆246Updated 3 years ago
• ukwa / webarchive-discovery
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆131Updated last month
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆441Updated last year
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• mementoweb / py-memento-client
  A Memento Client Library in Python
  ☆26Updated 7 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆47Updated 8 years ago
• pudo / normality
  A tiny library for Python text normalisation. Useful for ad-hoc text processing.
  ☆157Updated 3 months ago
• scrapy / protego
  A pure-Python robots.txt parser with support for modern conventions.
  ☆75Updated 2 weeks ago
• internetarchive / trough
  Trough: Big data, small databases.
  ☆41Updated last year
• niksite / url-normalize
  URL normalization for Python
  ☆99Updated 8 months ago
• harvard-lil / waczerciser
  Create and edit WARC and WACZ files
  ☆20Updated last year
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆31Updated last year
• sethmlarson / whatwg-url
  Python implementation of WHATWG URL Living Standard
  ☆21Updated last year
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆120Updated 2 months ago
• opensanctions / countrynames
  Utility library to turn country names into ISO two-letter codes
  ☆71Updated 4 months ago
• DocNow / diffengine
  track changes to the news, where news is anything with an RSS feed
  ☆179Updated 5 years ago
• mloesch / sickle
  Sickle: OAI-PMH for Humans
  ☆115Updated 2 years ago
• edsu / microdata
  python library for extracting html microdata
  ☆166Updated 2 years ago
• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆62Updated 6 years ago
• pykler / yajl-py
  Pure Python wrapper to the Yajl C Library
  ☆85Updated last year
• dhamaniasad / WARCTools
  A list of tools related to W(eb)ARC(hive)
  ☆66Updated 11 years ago
• WASAPI-Community / data-transfer-apis
  WASAPI data transfer APIs
  ☆48Updated 3 years ago
• archivesunleashed / notebooks
  Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archive…
  ☆26Updated 3 years ago
• legoktm / toolforge
  [OBSOLETE] Replaced by https://gitlab.wikimedia.org/toolforge-repos/python-toolforge
  ☆22Updated 2 years ago
• ukwa / ukwa-pywb
  ☆11Updated last month
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆155Updated 2 months ago
• seomoz / url-py
  URL Transformation, Sanitization
  ☆103Updated last year
• vhf / confusable_homoglyphs
  ϲοｎｆｕѕаｂｌе＿һοｍоɡｌｙｐｈｓ
  ☆162Updated last year
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆63Updated 4 months ago