Alternatives and similar repositories for deduplication

Users that are interested in deduplication are comparing it to the libraries listed below

• barseghyanartur / radar
  Generate random date(time) in Python.
  ☆10Updated last year
• browsermt / marian-dev
  Fast Neural Machine Translation in C++ - development repository
  ☆19Updated last year
• alvarobartt / opentrain
  🚂 Fine-tune OpenAI models for text classification, question answering, and more
  ☆16Updated 2 years ago
• lemurproject / Lucindri
  Indri search implementation on top of Lucene search engine
  ☆34Updated last year
• simonw / sqlite-fts4
  Custom Python functions for working with SQLite FTS4
  ☆22Updated 2 years ago
• sureshdsk / rich-diff
  CLI based diff viewer
  ☆23Updated 3 years ago
• scrapy / xtractmime
  https://mimesniff.spec.whatwg.org/ implementation for Python
  ☆13Updated last year
• gdamdam / sumo
  Tool to extracts the text from a web article urls and get frequency words, entities recognition, automatic summary and more
  ☆20Updated 6 years ago
• daveshap / GibberishDetector
  Detecting gibberish as a type of sentiment analysis with GPT2
  ☆24Updated 4 years ago
• adbar / py3langid
  Faster, modernized fork of the language identification tool langid.py
  ☆55Updated 5 months ago
• scrapy / pypydispatcher
  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  ☆16Updated 7 years ago
• pudo / banal
  Commons of stupid, simple Python micro functions. Pull requests very welcome.
  ☆19Updated last month
• dragnet-org / dragnet_data
  code and data used to build a training dataset for dragnet models
  ☆10Updated 4 years ago
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated 2 years ago
• WojciechMula / pyDAWG
  Python module (C extension and plain python) implementing DAWG
  ☆20Updated 3 years ago
• MontFerret / pyfer
  Python wrapper for Ferret
  ☆41Updated 3 years ago
• fujimotos / TinyFastSS
  An index data structure for approximate string search.
  ☆23Updated 6 years ago
• iaramer / dobbi
  An open-source NLP library: fast text cleaning and preprocessing
  ☆23Updated 3 years ago
• scrapinghub / shublang
  Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
  ☆16Updated 10 months ago
• allenai / cached_path
  A file utility for accessing both local and remote files through a unified interface.
  ☆41Updated last week
• explosion / wikid
  Generate a SQLite database from Wikipedia & Wikidata dumps.
  ☆35Updated last year
• bentoml / sentence-embedding-bento
  Sentence Embedding as a Service
  ☆15Updated last year
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆35Updated 2 weeks ago
• webrecorder / browsertrix-behaviors
  Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.
  ☆40Updated 2 weeks ago
• SkBlaz / tax2vec
  Interpretable feature construction from taxonomies for text classification
  ☆18Updated 3 years ago
• koursaros-ai / microservices
  Neural Elastic Inference and Search
  ☆19Updated 5 years ago
• microsoft / reconner
  ReconNER, Debug annotated Named Entity Recognition (NER) data for inconsistencies and get insights on improving the quality of your data.
  ☆35Updated 4 years ago
• cortexlabs / nucleus
  Cortex-compatible model server for Python and TensorFlow
  ☆17Updated 2 years ago
• thombashi / typepy
  A Python library for variable type checker/validator/converter at a run time.
  ☆16Updated 4 months ago
• sre-yield / sre-yield
  Python module to generate regular all expression matches
  ☆18Updated 2 years ago