Alternatives and similar repositories for deduplication

Users that are interested in deduplication are comparing it to the libraries listed below

• huridocs / pdf-reading-order
  ☆15Updated last year
• lemurproject / Lucindri
  Indri search implementation on top of Lucene search engine
  ☆35Updated last year
• nicolay-r / AREkit
  Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML
  ☆65Updated 10 months ago
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated 2 years ago
• iaramer / dobbi
  An open-source NLP library: fast text cleaning and preprocessing
  ☆23Updated 4 years ago
• AlonEirew / wikipedia-to-elastic
  Analyze and extract Wikipedia article text and attributes and store them into an ElasticSearch index or to json files (multilingual suppo…
  ☆47Updated 2 years ago
• alea-institute / nupunkt
  Next-generation Punkt sentence boundary detection with zero dependencies
  ☆24Updated 3 weeks ago
• pd3f / pd3f-core
  📑 Python Package to reconstruct the original continuous text from PDFs with language models
  ☆32Updated 2 years ago
• MBAigner / PDFSegmenter
  This library builds a graph-representation of the content of PDFs. The graph is then clustered, resulting page segments are classified an…
  ☆23Updated 5 years ago
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆124Updated last month
• adbar / py3langid
  Faster, modernized fork of the language identification tool langid.py
  ☆61Updated last year
• nlp-uoregon / famie
  FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction
  ☆24Updated 3 years ago
• microsoft / ARXGEN
  Scripts to parse arxiv documents for NLP tasks
  ☆19Updated 2 years ago
• PathwayCommons / semantic-search
  A simple semantic search engine for scientific papers.
  ☆28Updated 2 years ago
• hora-search / horapy
  🐍 Python bidding for the Hora Approximate Nearest Neighbor Search Algorithm library
  ☆73Updated 4 years ago
• webis-de / summary-explorer
  Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.
  ☆45Updated last year
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 8 years ago
• wanasit / chrono-python
  A natural language date parser. (Python version of chrono.js)
  ☆25Updated 6 months ago
• LanguageMachines / LuigiNLP
  A workflow system for Natural Language Processing.
  ☆21Updated 6 years ago
• allenai / cached_path
  A file utility for accessing both local and remote files through a unified interface.
  ☆44Updated 2 months ago
• maxdotio / neural-solr
  Neural Solr = Solr 9 + Mighty Inference + Node
  ☆18Updated 3 years ago
• dragnet-org / dragnet_data
  code and data used to build a training dataset for dragnet models
  ☆10Updated 5 years ago
• coastalcph / lexlms
  LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
  ☆20Updated 2 years ago
• koursaros-ai / microservices
  Neural Elastic Inference and Search
  ☆19Updated 6 years ago
• MontFerret / pyfer
  Python wrapper for Ferret
  ☆45Updated 3 years ago
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆25Updated last year
• DanielJDufour / date-extractor
  Extract dates from text
  ☆66Updated 4 years ago
• codogogo / towerparse
  Tower Parse: Low-Resource Dependency Parsing via Hierarchical Source Selection
  ☆15Updated 4 years ago
• onfido / faiss_prebuilt
  Prebuilt .whl files for MacOS + Linux of the Facebook FAISS library
  ☆56Updated 3 years ago
• bentoml / sentence-embedding-bento
  Sentence Embedding as a Service
  ☆15Updated 5 months ago