Alternatives and similar repositories for urlcanon

Users that are interested in urlcanon are comparing it to the libraries listed below

• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆42Updated last year
• internetarchive / warc
  Python library for reading and writing warc files
  ☆244Updated 3 years ago
• edsu / microdata
  python library for extracting html microdata
  ☆167Updated 2 years ago
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆117Updated 9 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 3 years ago
• ukwa / webarchive-discovery
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆128Updated last month
• niksite / url-normalize
  URL normalization for Python
  ☆97Updated 4 months ago
• pudo / normality
  A tiny library for Python text normalisation. Useful for ad-hoc text processing.
  ☆153Updated last month
• internetarchive / trough
  Trough: Big data, small databases.
  ☆42Updated last year
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆62Updated 3 weeks ago
• DocNow / diffengine
  track changes to the news, where news is anything with an RSS feed
  ☆179Updated 5 years ago
• TeamHG-Memex / soft404
  A classifier for detecting soft 404 pages
  ☆56Updated 2 years ago
• scrapy / protego
  A pure-Python robots.txt parser with support for modern conventions.
  ☆70Updated last month
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆429Updated 8 months ago
• TeamHG-Memex / extract-html-diff
  extract difference between two html pages
  ☆32Updated 7 years ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆134Updated 5 years ago
• seomoz / url-py
  URL Transformation, Sanitization
  ☆103Updated last year
• sethmlarson / whatwg-url
  Python implementation of WHATWG URL Living Standard
  ☆21Updated last year
• seomoz / reppy
  Modern robots.txt Parser for Python
  ☆196Updated last year
• TeamHG-Memex / undercrawler
  A generic crawler
  ☆78Updated 7 years ago
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 7 years ago
• mloesch / sickle
  Sickle: OAI-PMH for Humans
  ☆113Updated 2 years ago
• harvard-lil / waczerciser
  Create and edit WARC and WACZ files
  ☆15Updated 8 months ago
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 2 years ago
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆152Updated last month
• archivesunleashed / notebooks
  Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archive…
  ☆26Updated 2 years ago
• opensanctions / countrynames
  Utility library to turn country names into ISO two-letter codes
  ☆70Updated 3 weeks ago
• barseghyanartur / tld
  Extracts the top level domain (TLD) from the URL given.
  ☆180Updated 3 months ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆56Updated last month