Alternatives and similar repositories for deduplication:

Users that are interested in deduplication are comparing it to the libraries listed below

• rgov / sqlite_protobuf
  A SQLite extension for extracting values from serialized Protobuf messages
  ☆87Updated last month
• pascaldekloe / flit
  fixed-length integer trim
  ☆33Updated 2 years ago
• malbrain / HatTrie
  A compact implementation of Dr. Askitis HatTrie
  ☆80Updated 10 years ago
• fd0 / rabin-cdc
  Fast implementation of Content Defined Chunking (CDC) based on a rolling Rabin Checksum in C.
  ☆53Updated 10 years ago
• cruzdb / cruzdb
  Append-only key-value database on a distributed shared-log
  ☆49Updated 7 months ago
• nlfiedler / fastcdc-rs
  FastCDC implementation in Rust
  ☆149Updated last year
• ronomon / direct-io
  Direct IO helpers for block devices and regular files on FreeBSD, Linux, macOS and Windows.
  ☆72Updated 2 years ago
• wolfgarbe / EliasFanoCompression
  EliasFanoCompression: quasi-succinct compression of sorted integers in C#
  ☆45Updated 3 years ago
• fastly / uslab
  Lock-free slab allocator / freelist.
  ☆65Updated 9 years ago
• willemt / bmon
  Batch Monitor - Gain performance by combining work from multiple threads into a single batch
  ☆30Updated 9 years ago
• jedisct1 / cpace
  A CPace PAKE implementation using libsodium.
  ☆35Updated 4 years ago
• patrickfrey / strus
  Library implementing the storage and the query evaluation for a text search engine. It uses on a key value store database interface to st…
  ☆47Updated 3 years ago
• srijs / rust-gearhash
  Fast, SIMD-accelerated hash function for content-defined chunking
  ☆23Updated 4 years ago
• hse-project / mpool
  mpool UAPI and CLI for HSE 1.x
  ☆36Updated 3 years ago
• spoke-d / thermionic
  SQLite Clustered Database
  ☆18Updated 5 years ago
• LumoSQL / lumosql
  Official Github Mirror of the LumoSQL Database Project (https://lumosql.org/src/lumosql)
  ☆180Updated last year
• fanf2 / hg64
  a 64-bit histogram / quantile sketch
  ☆58Updated 2 months ago
• LMDB / memcachedb
  memcachedb ported from BerkeleyDB to LMDB originally from http://memcachedb.googlecode.com/svn/trunk
  ☆86Updated 9 years ago
• wangyi-fudan / wyHLL
  The dream accurate approximate set cardinality estimator based on 3-bit HyperLogLog. More accurate than Redis HyperLogLog.
  ☆55Updated 4 years ago
• WebDeltaSync / WebR2sync_plus
  WebR2sync+
  ☆12Updated 8 years ago
• pmwkaa / ioarena
  Embedded storage benchmarking tool
  ☆138Updated 2 years ago
• vkrasnov / avx_qsort
  Quick sort code using AVX2 instructions
  ☆68Updated 7 years ago
• jkominek / fdbfs
  A not-yet-ready-for-use FoundationDB-backed FUSE filesystem. Seriously, don't use it.
  ☆13Updated 11 months ago
• fidlej / idzip
  Automatically exported from code.google.com/p/idzip
  ☆40Updated 10 years ago
• jsnell / parallel-xxhash
  Compute xxHash hash codes for 8 keys in parallel
  ☆46Updated 5 years ago
• fredrikwidlund / pounce
  HTTP benchmark utility
  ☆21Updated last year
• btrask / lsmdb
  Proof of concept LSM-tree built on MDB
  ☆17Updated 9 years ago
• gtoubassi / femtozip
  FemtoZip is a "shared dictionary" compression library optimized for small documents that may not compress well with traditional tools suc…
  ☆145Updated 3 years ago
• pmwkaa / engine.so
  Tracking, Benchmarking and Sharing Information about an open source embedded data storage engines, internals, architectures, data storage…
  ☆181Updated 6 years ago
• benwebber / sqlite3-uuid
  SQLite extension for generating UUIDs
  ☆47Updated 6 years ago