Alternatives and similar repositories for dedup

Users that are interested in dedup are comparing it to the libraries listed below

• transducens / linguacrawl
  Crawling engine that crawls a set of top-level domains looking for documents in a list of languages
  ☆11Updated last year
• jonathandunn / common_crawl_corpus
  Scripts for building a geo-located web corpus using Common Crawl data
  ☆11Updated 4 months ago
• flexudy-pipe / sentence-doctor
  Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …
  ☆61Updated 4 years ago
• dataiku / dss-plugin-nlp-preparation
  Dataiku DSS plugin to detect languages, correct misspellings, and clean text data 🧼
  ☆22Updated 7 months ago
• alexyorke / butter-fingers
  A python library to generate highly realistic typos (fuzz-testing)
  ☆12Updated 5 months ago
• TurkuNLP / ocr-correction
  Post-processing OCR errors with seq2seq models
  ☆28Updated 5 years ago
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated 2 years ago
• TyMick / loan-risk-neural-network
  Loan Risk Prediction Neural Network and API
  ☆17Updated 4 years ago
• kensk8er / langdist
  Multilingual Language Modeling Toolkit
  ☆11Updated 8 years ago
• staeiou / arxiv_archive
  A tidy and complete archive of metadata for papers on arxiv.org, 1993-2019
  ☆28Updated 5 years ago
• fionapigott / conversation-builder
  Python script to assemble individual Tweets from a public Twitter stream (either Gnip activity-streams format or original Twitter API for…
  ☆12Updated 9 years ago
• explosion / spacy-vectors-builder
  🌸 Train floret vectors
  ☆18Updated 2 years ago
• JonathanRaiman / epub_conversion
  Python package for converting xml and epubs to text files
  ☆33Updated 5 years ago
• amansrivastava17 / text-preprocess-python
  Text preprocessing tools in python.
  ☆27Updated 7 years ago
• societe-generale / core-nlg
  CoreNLG is an easy to use and productivity oriented Python library for Natural Language Generation. It aims to provide the essential tool…
  ☆27Updated 4 years ago
• wroberts / fsed
  Aho-Corasick string replacement utility
  ☆24Updated 5 years ago
• xigt / freki
  Analyze XML extracted from PDFs (e.g. from TET or PDFMiner)
  ☆20Updated 7 years ago
• Helsinki-NLP / sentimentator
  Tool for sentiment analysis annotation
  ☆13Updated 5 months ago
• derekgreene / topicscan
  TopicScan: Visualization and validation interface for NMF Topic Modeling
  ☆23Updated 5 years ago
• sahyagiri / DistinctKeywords
  semantically distinct key phrase extraction using hilbert hashes.
  ☆50Updated 3 years ago
• dedupeio / doublemetaphone
  Python wrapper for a C++ Double Metaphone
  ☆15Updated last week
• vincent9514 / Text-Variant-Generation
  📄Neural Sentential Paraphrase Generation to Augment Chatbot Training Dataset
  ☆21Updated 2 years ago
• maxent-ai / zeroshot_topics
  Topic Inference with Zeroshot models
  ☆61Updated 2 years ago
• boberle / dependency2tree
  Convert CoNLL output of a dependency parser into a latex or graphviz tree
  ☆12Updated 5 years ago
• libindic / soundex
  Soundex Phonetic Code Algorithm Demo for Indian Languages. Supports all indian languages and English. Provides intra-indic string compari…
  ☆58Updated 6 years ago
• domyounglee / TF-TrigramBlocking-transformer
  Transformer based Trigram Blocking implementation in Tensorflow
  ☆11Updated 5 years ago
• pmbaumgartner / spacy-setfit-textcat
  ☆30Updated 3 years ago
• BramVanroy / spacy-extreme
  An example of how to use spaCy for extremely large files without running into memory issues
  ☆36Updated 2 years ago
• oudalab / Arabic-NER
  ☆32Updated 7 years ago
• projecte-aina / spacy
  Pre-production releases for Spacy in Catalan
  ☆14Updated 3 years ago