Alternatives and similar repositories for dedup

Users that are interested in dedup are comparing it to the libraries listed below

• transducens / linguacrawl
  Crawling engine that crawls a set of top-level domains looking for documents in a list of languages
  ☆11Updated last year
• flexudy-pipe / sentence-doctor
  Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …
  ☆61Updated 5 years ago
• alexyorke / butter-fingers
  A python library to generate highly realistic typos (fuzz-testing)
  ☆12Updated 7 months ago
• bhaddow / pmindia-crawler
  Code for extracting parallel corpora from pmindia
  ☆16Updated 5 years ago
• Helsinki-NLP / sentimentator
  Tool for sentiment analysis annotation
  ☆13Updated 7 months ago
• openredact / nerwhal
  This is a prototype of a multi-lingual suite for named-entity recognition in Python.
  ☆21Updated last year
• amansrivastava17 / text-preprocess-python
  Text preprocessing tools in python.
  ☆27Updated 7 years ago
• ad-freiburg / tokenization-repair
  Correction of spaces with character-based neural language models.
  ☆13Updated 3 years ago
• laugustyniak / textlytics
  Text processing library for sentiment analysis and related tasks
  ☆27Updated 7 years ago
• BramVanroy / spacy-extreme
  An example of how to use spaCy for extremely large files without running into memory issues
  ☆36Updated 3 years ago
• ilinguistics / common_crawl_corpus
  Scripts for building a geo-located web corpus using Common Crawl data
  ☆11Updated last week
• pmbaumgartner / spacy-setfit-textcat
  ☆30Updated 3 years ago
• TurkuNLP / wikibert
  BERT models for many languages created from Wikipedia texts
  ☆33Updated 5 years ago
• DanielJDufour / date-extractor
  Extract dates from text
  ☆65Updated 4 years ago
• bedapudi6788 / txt2txt
  Extremely easy to use sequence to sequence library with attention, for text to text conversion tasks.
  ☆39Updated 5 years ago
• TurkuNLP / ocr-correction
  Post-processing OCR errors with seq2seq models
  ☆28Updated 5 years ago
• JarrodAJ / sec_employee_information_extraction
  NSS Capstone project to use natural language modeling, classification, and information extraction to get the exact employee count values …
  ☆15Updated 7 years ago
• explosion / spacy-vectors-builder
  🌸 Train floret vectors
  ☆18Updated 2 years ago
• openlegaldata / legal-ner
  Named entity recognition for the legal domain
  ☆42Updated 4 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 5 years ago
• krandiash / gpt3-nli
  Training a model without a dataset for natural language inference (NLI)
  ☆25Updated 5 years ago
• nicolay-r / AREkit
  Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML
  ☆65Updated 9 months ago
• bitextor / bifixer
  Tool to fix bitexts and tag near-duplicates for removal
  ☆33Updated 2 months ago
• Data4Democracy / hate_speech_detector
  ☆35Updated 2 years ago
• argilla-io / biome-text
  Custom Natural Language Processing with big and small models 🌲🌱
  ☆66Updated 4 years ago
• sahyagiri / DistinctKeywords
  semantically distinct key phrase extraction using hilbert hashes.
  ☆50Updated 3 years ago
• vincent9514 / Text-Variant-Generation
  📄Neural Sentential Paraphrase Generation to Augment Chatbot Training Dataset
  ☆21Updated 2 years ago
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• stephantul / unitoken
  Tokenization across languages. Useful as preprocessing for subword tokenization.
  ☆21Updated 2 years ago
• dataiku / dss-plugin-nlp-preparation
  Dataiku DSS plugin to detect languages, correct misspellings, and clean text data 🧼
  ☆22Updated 9 months ago