Marcnuth / deduplicationLinks
Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆19Updated 2 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below
Sorting:
- ☆15Updated last year
- This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…☆34Updated 2 years ago
- An open-source NLP library: fast text cleaning and preprocessing☆23Updated 4 years ago
- This library builds a graph-representation of the content of PDFs. The graph is then clustered, resulting page segments are classified an…☆23Updated 5 years ago
- Complex data extraction and orchestration framework designed for processing unstructured documents. It integrates AI-powered document pip…☆80Updated this week
- Indri search implementation on top of Lucene search engine☆35Updated last year
- 📑 Python Package to reconstruct the original continuous text from PDFs with language models☆32Updated 2 years ago
- 🚂 Fine-tune OpenAI models for text classification, question answering, and more☆17Updated 2 years ago
- Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML☆65Updated last year
- Documentation effort for the BookCorpus dataset☆34Updated 4 years ago
- A simple semantic search engine for scientific papers.☆28Updated 2 years ago
- Efficiently search the most similar strings against the query in Python.☆18Updated 8 months ago
- code and data used to build a training dataset for dragnet models☆10Updated 5 years ago
- An example of graph embeddings for wikipedia page recommendations☆11Updated 4 years ago
- Translate HTML using Argos Translate☆57Updated 2 years ago
- 🖍️ Highlight text in documents☆111Updated 9 months ago
- ☆69Updated 5 years ago
- Custom Python functions for working with SQLite FTS4☆23Updated 3 years ago
- Neural Solr = Solr 9 + Mighty Inference + Node☆18Updated 3 years ago
- 🐍 Python bidding for the Hora Approximate Nearest Neighbor Search Algorithm library☆73Updated 4 years ago
- Extract dates from text☆66Updated 5 years ago
- ☆20Updated 4 years ago
- Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.☆45Updated last year
- Fast edit distance Python extension written in Cython/C++. Supports Levenshtein distance and Damerau Optimal String Alignment (OSA) dista…☆25Updated 8 months ago
- Tools to construct and process Common Crawl webgraphs☆105Updated last week
- Tool to extracts the text from a web article urls and get frequency words, entities recognition, automatic summary and more☆20Updated 7 years ago
- Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.☆66Updated last week
- Python based Wikidata framework for easy dataframe extraction☆45Updated 2 years ago
- Yet another tool to search through your (exported) ChatGPT conversations☆13Updated last month
- Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …☆62Updated 5 years ago