fake-name / IntraArchiveDeduplicatorLinks
Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
☆104Updated 2 years ago
Alternatives and similar repositories for IntraArchiveDeduplicator
Users that are interested in IntraArchiveDeduplicator are comparing it to the libraries listed below
Sorting:
- Serving content from a WARC☆62Updated 12 years ago
- ☆89Updated last year
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆46Updated 7 years ago
- Github-Repository of the pHash.org library for perceptual hashing.☆222Updated 7 years ago
- Implementation of perceptual image hash calculation in Python☆132Updated last year
- Python library for reading and writing warc files☆245Updated 3 years ago
- WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.☆47Updated 7 years ago
- Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io☆38Updated 9 years ago
- isk-daemon is an open source standalone server and library capable of adding content-based (visual) image searching to any image related …☆140Updated 10 years ago
- A queue-controlled browser automation tool for improving web crawl quality☆62Updated last month
- Rewriting web proxy and archival tool. At this point, it just tries to download all the things.☆202Updated this week
- bktree data structure with a Python interface for a CPP implementation☆13Updated 8 years ago
- Detect source resolution of upscaled images☆290Updated 2 months ago
- An HTTP-based warc-to-zip converter☆12Updated 12 years ago
- Tool to detect (and get rid of) similar images using perceptual hashing (pHash lib)☆83Updated 8 years ago
- Remote client for distributed automated HTTP(s) content fetching.☆77Updated last week
- Saves proxied HTTP traffic to a WARC file.☆30Updated 11 years ago
- Nondestructive warc-in-tar to warc conversion☆27Updated 12 years ago
- A list of tools related to W(eb)ARC(hive)☆64Updated 10 years ago
- Python library to calculate the difference hash (perceptual hash) for a given image, useful for detecting duplicates☆370Updated last year
- This is a side project from 2008. This package contains a tool for automatically cropping and deskewing images of book pages captured by …☆28Updated 12 years ago
- A Python Perceptual Image Hashing Module☆214Updated 3 years ago
- Web archiving using Google Chrome☆47Updated 5 years ago
- C++ implementation of hamming distance algorithm HmSearch using Kyoto Cabinet☆42Updated 9 years ago
- A slim, non-SWIG Python adapter to CTesseract (Tesseract OCR for C).☆24Updated 11 years ago
- Python BK-tree data structure to allow fast querying of "close" matches☆186Updated 3 years ago
- 📂🛡️Suite of tools for file fixity (data protection for long term storage⌛) using redundant error correcting codes, hash auditing and du…☆147Updated last month
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆50Updated 3 weeks ago
- Naïve Bayesian Text Classifier on Redis☆116Updated 6 years ago
- Implement SQLite table-valued functions with Python☆59Updated 2 years ago