Alternatives and similar repositories for dedup:

Users that are interested in dedup are comparing it to the libraries listed below

• sahyagiri / DistinctKeywords
  semantically distinct key phrase extraction using hilbert hashes.
  ☆48Updated 3 years ago
• dataiku / dss-plugin-nlp-preparation
  Dataiku DSS plugin to detect languages, correct misspellings, and clean text data 🧼
  ☆22Updated 3 months ago
• Marcnuth / deduplication
  Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
  ☆18Updated last year
• cyrilou242 / learning-lightnr
  Generate multiple choice fill-in-the-blank questions from any article.
  ☆13Updated 2 years ago
• jonathandunn / common_crawl_corpus
  Scripts for building a geo-located web corpus using Common Crawl data
  ☆11Updated 2 weeks ago
• transducens / linguacrawl
  Crawling engine that crawls a set of top-level domains looking for documents in a list of languages
  ☆10Updated last year
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• TurkuNLP / ocr-correction
  Post-processing OCR errors with seq2seq models
  ☆28Updated 4 years ago
• TurkuNLP / wikibert
  BERT models for many languages created from Wikipedia texts
  ☆33Updated 4 years ago
• bhaddow / pmindia-crawler
  Code for extracting parallel corpora from pmindia
  ☆16Updated 5 years ago
• YungChunLu / Entity-Resolution-On-Text-Similarity
  In this project, we need to find out commercial products listed on Google that refer to the same entity across Amazon by comparing the si…
  ☆11Updated 8 years ago
• flexudy-pipe / sentence-doctor
  Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …
  ☆61Updated 4 years ago
• tm4roon / survey
  Survey on machine learning.
  ☆14Updated 4 years ago
• domyounglee / TF-TrigramBlocking-transformer
  Transformer based Trigram Blocking implementation in Tensorflow
  ☆11Updated 5 years ago
• akshayakp97 / TL-DR
  An end-to-end event extraction and summarization system.
  ☆21Updated 4 years ago
• lemay-ai / lazyTextPredict
  Text classification automl
  ☆21Updated 3 years ago
• karndeb / NLP-Service
  ☆13Updated 3 years ago
• caltechlibrary / documentarist
  Process Caltech Archives' digital documents and photos, and annotate each page or image with information about its contents
  ☆12Updated 2 years ago
• iaramer / dobbi
  An open-source NLP library: fast text cleaning and preprocessing
  ☆23Updated 3 years ago
• alexyorke / butter-fingers
  A python library to generate highly realistic typos (fuzz-testing)
  ☆11Updated last month
• richardxiong / dnnQuery
  Deep neural parser for database query
  ☆18Updated 2 years ago
• fujimotos / TinyFastSS
  An index data structure for approximate string search.
  ☆23Updated 5 years ago
• RxNLP / nlp-cloud-apis
  RxNLP APIs for clustering sentences, extracting topics, counting words & n-grams, extracting text from html or URL, computing similarity …
  ☆15Updated 5 years ago
• domyounglee / Transformer-Summarization
  An optimized Transformer based abstractive summarization model with Tensorflow
  ☆16Updated 2 years ago
• text-machine-lab / TEA
  Extracting narrative timelines (i.e. order and timing of events) from text
  ☆20Updated 6 years ago
• UKPLab / incorporating-relevance
  Code for "Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking" (https://arxiv.org/abs/2…
  ☆13Updated 2 years ago
• talmago / spacy_ke
  Keyword extraction with spaCy
  ☆31Updated 3 years ago
• facebookresearch / DataEfficientNLG
  NLG Best Practices for Data-Efficient Modeling How to Train Production-Ready Models with Little Data
  ☆10Updated 3 years ago
• patil-suraj / distillbart-mnli
  No Teacher BART distillation experiment for NLI tasks
  ☆26Updated 4 years ago
• epfml / Bi-Sent2Vec
  Robust Cross-lingual Embeddings from Parallel Sentences
  ☆22Updated 4 years ago