Alternatives and similar repositories for deduplication:

Users that are interested in deduplication are comparing it to the libraries listed below

• karlb / sqlite-spellfix
  Loadable spellfix1 extension for sqlite as python package
  ☆26Updated 9 months ago
• N0taN3rd / simplechrome
  Webrecorders DevTools Protocol Automation Library
  ☆17Updated 2 years ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 7 years ago
• iaramer / dobbi
  An open-source NLP library: fast text cleaning and preprocessing
  ☆23Updated 3 years ago
• lemurproject / Lucindri
  Indri search implementation on top of Lucene search engine
  ☆34Updated 11 months ago
• dragnet-org / dragnet_data
  code and data used to build a training dataset for dragnet models
  ☆10Updated 4 years ago
• maxdotio / neural-solr
  Neural Solr = Solr 9 + Mighty Inference + Node
  ☆16Updated 2 years ago
• zyocum / dedup
  Find duplicate text files.
  ☆13Updated last month
• SkBlaz / tax2vec
  Interpretable feature construction from taxonomies for text classification
  ☆18Updated 2 years ago
• microsoft / reconner
  ReconNER, Debug annotated Named Entity Recognition (NER) data for inconsistencies and get insights on improving the quality of your data.
  ☆34Updated 4 years ago
• SYSTRAN / fuzzy-match
  Library and command line utility to do approximate string matching of a source against a bitext index and get matched source and target.
  ☆47Updated last month
• poke1024 / simtrie
  An efficient data structure for fast string similarity searches
  ☆22Updated 4 years ago
• jackbandy / bookcorpus-datasheet
  Documentation effort for the BookCorpus dataset
  ☆33Updated 3 years ago
• barseghyanartur / radar
  Generate random date(time) in Python.
  ☆10Updated 11 months ago
• webrecorder / pywb-remote-browsers
  Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives
  ☆14Updated 3 years ago
• scrapy / xtractmime
  https://mimesniff.spec.whatwg.org/ implementation for Python
  ☆14Updated last year
• go-air / dupi
  A tool to find all duplicates in large sets of text documents.
  ☆16Updated 3 years ago
• comet-ml / blog-serving-hugging-face-models
  ☆20Updated 3 years ago
• nicolay-r / AREkit
  Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML
  ☆62Updated last month
• simonw / datasette-search-all
  Datasette plugin for searching all searchable tables at once
  ☆22Updated 5 months ago
• EleutherAI / tokengrams
  Efficiently computing & storing token n-grams from large corpora
  ☆18Updated 4 months ago
• internetarchive / sandcrawler
  Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
  ☆25Updated 6 months ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• jina-ai / executor-hnsw-postgres
  A production-ready, scalable Indexer for the Jina neural search framework, based on HNSW and PSQL
  ☆29Updated 2 years ago
• wanghaisheng / awesome-web-data-extractor
  A curated list of promising Web Data Extractors resources
  ☆28Updated 5 years ago
• allenai / cached_path
  A file utility for accessing both local and remote files through a unified interface.
  ☆37Updated last month
• scrapy / pypydispatcher
  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  ☆16Updated 7 years ago
• stephantul / unitoken
  Tokenization across languages. Useful as preprocessing for subword tokenization.
  ☆22Updated 2 years ago
• hiredscorelabs / seqtolang
  Multi-Langauge Identification
  ☆29Updated 6 months ago