iipc / urlcanon

Related projects ⓘ

Alternatives and complementary repositories for urlcanon

• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆40Updated 3 months ago
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆100Updated this week
• internetarchive / trough
  Trough: Big data, small databases.
  ☆40Updated 3 months ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆42Updated 6 years ago
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆21Updated last week
• opensanctions / countrynames
  Utility library to turn country names into ISO two-letter codes
  ☆66Updated this week
• nla / outbackcdx
  Web archive index server based on RocksDB
  ☆32Updated this week
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 2 years ago
• TeamHG-Memex / soft404
  A classifier for detecting soft 404 pages
  ☆57Updated last year
• TeamHG-Memex / MaybeDont
  A component that tries to avoid downloading duplicate content
  ☆27Updated 6 years ago
• scrapy-plugins / scrapy-streaming
  ☆18Updated 8 years ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆117Updated 3 months ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆50Updated last month
• scrapy / protego
  A pure-Python robots.txt parser with support for modern conventions.
  ☆55Updated this week
• DocNow / waybackprov
  utility to fetch provenance information from Internet Archive's Wayback Machine
  ☆13Updated 2 years ago
• oduwsdl / MemGator
  A Memento Aggregator CLI and Server in Go
  ☆57Updated 6 months ago
• nlnwa / warchaeology
  Command line tool for digging into WARC files
  ☆34Updated 3 weeks ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆60Updated 4 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆102Updated last week
• internetarchive / arch
  Web application for distributed compute analysis of Archive-It web archive collections.
  ☆15Updated 2 months ago
• scrapy / scurl
  Performance-focused replacement for Python urllib
  ☆21Updated 6 years ago
• Apoc2400 / Reftag
  Wikipedia citation tool for Google Books, New York Times, ISBN, DOI and more
  ☆21Updated 8 years ago
• edsu / microdata
  python library for extracting html microdata
  ☆165Updated last year
• ArchiveTeam / NewsGrabber
  Grabbing all news.
  ☆62Updated 4 years ago
• scrapinghub / shublang
  Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
  ☆15Updated 4 months ago
• scrapy-plugins / scrapy-pagestorage
  A scrapy extension to store requests and responses information in storage service
  ☆26Updated 2 years ago
• ukwa / ukwa-pywb
  ☆11Updated last year
• scrapy / xtractmime
  https://mimesniff.spec.whatwg.org/ implementation for Python
  ☆14Updated 10 months ago
• TeamHG-Memex / extract-html-diff
  extract difference between two html pages
  ☆32Updated 6 years ago
• internetarchive / sandcrawler
  Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
  ☆25Updated 3 months ago