Alternatives and similar repositories for dedup

Users that are interested in dedup are comparing it to the libraries listed below

• transducens / linguacrawl
  Crawling engine that crawls a set of top-level domains looking for documents in a list of languages
  ☆11Updated last year
• ilinguistics / common_crawl_corpus
  Scripts for building a geo-located web corpus using Common Crawl data
  ☆11Updated last month
• flexudy-pipe / sentence-doctor
  Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …
  ☆62Updated 5 years ago
• staeiou / arxiv_archive
  A tidy and complete archive of metadata for papers on arxiv.org, 1993-2019
  ☆28Updated 5 years ago
• alexyorke / butter-fingers
  A python library to generate highly realistic typos (fuzz-testing)
  ☆13Updated 8 months ago
• ines / spacy-streamlit-demo
  ☆19Updated 3 years ago
• openredact / nerwhal
  This is a prototype of a multi-lingual suite for named-entity recognition in Python.
  ☆21Updated last year
• etalab-ia / piaf-code
  This repo contains the code used to generate the French Wikipedia sample used in the QA annotation project PIAF
  ☆11Updated 4 years ago
• TurkuNLP / wikibert
  BERT models for many languages created from Wikipedia texts
  ☆33Updated 5 years ago
• dataiku / dss-plugin-nlp-preparation
  Dataiku DSS plugin to detect languages, correct misspellings, and clean text data 🧼
  ☆22Updated 10 months ago
• Sirsirious / NLPTools
  A set of NLP tools created during my medium NLP Explanation series.
  ☆31Updated last year
• JonathanRaiman / epub_conversion
  Python package for converting xml and epubs to text files
  ☆33Updated 5 years ago
• pmbaumgartner / spacy-setfit-textcat
  ☆30Updated 3 years ago
• TurkuNLP / ocr-correction
  Post-processing OCR errors with seq2seq models
  ☆28Updated 5 years ago
• DanielJDufour / date-extractor
  Extract dates from text
  ☆66Updated 4 years ago
• domyounglee / TF-TrigramBlocking-transformer
  Transformer based Trigram Blocking implementation in Tensorflow
  ☆11Updated 5 years ago
• laugustyniak / textlytics
  Text processing library for sentiment analysis and related tasks
  ☆27Updated 7 years ago
• argilla-io / biome-text
  Custom Natural Language Processing with big and small models 🌲🌱
  ☆66Updated 4 years ago
• sahyagiri / DistinctKeywords
  semantically distinct key phrase extraction using hilbert hashes.
  ☆50Updated 3 years ago
• writer / fitbert
  Use BERT to Fill in the Blanks
  ☆84Updated 3 years ago
• webis-de / summary-explorer
  Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.
  ☆45Updated last year
• BramVanroy / spacy-extreme
  An example of how to use spaCy for extremely large files without running into memory issues
  ☆36Updated 3 years ago
• maxent-ai / zeroshot_topics
  Topic Inference with Zeroshot models
  ☆61Updated 2 years ago
• vincent9514 / Text-Variant-Generation
  📄Neural Sentential Paraphrase Generation to Augment Chatbot Training Dataset
  ☆21Updated 3 years ago
• TyMick / loan-risk-neural-network
  Loan Risk Prediction Neural Network and API
  ☆17Updated 5 years ago
• bond005 / deep_ner
  Named entity recognizer based on ELMo or BERT as feature extractor and CRF as final classifier
  ☆80Updated 2 years ago
• AlonEirew / wikipedia-to-elastic
  Analyze and extract Wikipedia article text and attributes and store them into an ElasticSearch index or to json files (multilingual suppo…
  ☆47Updated 2 years ago
• Marcnuth / deduplication
  Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
  ☆19Updated 2 years ago
• piEsposito / transformers-low-code-experiments
  Low-code pre-built pipelines for experiments with huggingface/transformers for Data Scientists in a rush.
  ☆16Updated 5 years ago
• p-sodmann / Spacy3Textcat
  SpacyV3 Text Categorizer Tutorial
  ☆17Updated 5 years ago