Alternatives and similar repositories for deduplication

Users that are interested in deduplication are comparing it to the libraries listed below

• huridocs / pdf-reading-order
  ☆15Updated last year
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated 2 years ago
• marieai / marie-ai
  Complex data extraction and orchestration framework designed for processing unstructured documents. It integrates AI-powered document pip…
  ☆80Updated this week
• alvarobartt / opentrain
  🚂 Fine-tune OpenAI models for text classification, question answering, and more
  ☆17Updated 2 years ago
• MBAigner / PDFSegmenter
  This library builds a graph-representation of the content of PDFs. The graph is then clustered, resulting page segments are classified an…
  ☆23Updated 5 years ago
• EleutherAI / openwebtext2
  ☆92Updated 3 years ago
• lemurproject / Lucindri
  Indri search implementation on top of Lucene search engine
  ☆35Updated last year
• iaramer / dobbi
  An open-source NLP library: fast text cleaning and preprocessing
  ☆23Updated 4 years ago
• gdamdam / sumo
  Tool to extracts the text from a web article urls and get frequency words, entities recognition, automatic summary and more
  ☆20Updated 6 years ago
• mammothb / editdistpy
  Fast edit distance Python extension written in Cython/C++. Supports Levenshtein distance and Damerau Optimal String Alignment (OSA) dista…
  ☆25Updated 7 months ago
• jackbandy / bookcorpus-datasheet
  Documentation effort for the BookCorpus dataset
  ☆34Updated 4 years ago
• michaelachmann / gpt-cost-estimator
  A cost estimator for OpenAI API calls in tqdm loops.
  ☆20Updated last year
• allenai / cached_path
  A file utility for accessing both local and remote files through a unified interface.
  ☆45Updated 3 weeks ago
• butlerlabs / docai
  DocAI helps developers quickly build document, image and text processing pipelines using open source and cloud-based machine learning mod…
  ☆20Updated 3 years ago
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆25Updated last year
• pd3f / pd3f-core
  📑 Python Package to reconstruct the original continuous text from PDFs with language models
  ☆32Updated 2 years ago
• dragnet-org / dragnet_data
  code and data used to build a training dataset for dragnet models
  ☆10Updated 5 years ago
• nicolay-r / AREkit
  Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML
  ☆65Updated 11 months ago
• nlp-uoregon / famie
  FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction
  ☆24Updated 3 years ago
• alea-institute / nupunkt
  Next-generation Punkt sentence boundary detection with zero dependencies
  ☆26Updated last month
• bentoml / sentence-embedding-bento
  Sentence Embedding as a Service
  ☆15Updated 6 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• Layout-Parser / annotation-service
  ☆20Updated 4 years ago
• webis-de / summary-explorer
  Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.
  ☆45Updated last year
• RepoAnalysis / RepoSnipy
  Neural search engine for discovering semantically similar Python repositories on GitHub
  ☆27Updated last year
• coastalcph / lexlms
  LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
  ☆20Updated 2 years ago
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆126Updated 2 months ago
• PathwayCommons / semantic-search
  A simple semantic search engine for scientific papers.
  ☆28Updated 2 years ago
• comet-ml / blog-serving-hugging-face-models
  ☆20Updated 4 years ago
• neuml / txtmarker
  🖍️ Highlight text in documents
  ☆110Updated 8 months ago