Marcnuth / deduplicationLinks
Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆19Updated 2 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below
Sorting:
- ☆15Updated last year
- This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…☆34Updated 2 years ago
- 📑 Python Package to reconstruct the original continuous text from PDFs with language models☆32Updated 2 years ago
- Scripts to parse arxiv documents for NLP tasks☆19Updated 2 years ago
- A robust web archive analytics toolkit☆129Updated 3 months ago
- This library builds a graph-representation of the content of PDFs. The graph is then clustered, resulting page segments are classified an…☆23Updated 5 years ago
- An open-source NLP library: fast text cleaning and preprocessing☆23Updated 4 years ago
- Indri search implementation on top of Lucene search engine☆35Updated last year
- FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction☆24Updated 3 years ago
- Faster, modernized fork of the language identification tool langid.py☆60Updated last year
- ☆94Updated 3 years ago
- An example of graph embeddings for wikipedia page recommendations☆11Updated 4 years ago
- Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.☆66Updated last week
- A workflow system for Natural Language Processing.☆21Updated 6 years ago
- Complex data extraction and orchestration framework designed for processing unstructured documents. It integrates AI-powered document pip…☆80Updated this week
- Tools to construct and process Common Crawl webgraphs☆105Updated last week
- 🚂 Fine-tune OpenAI models for text classification, question answering, and more☆17Updated 2 years ago
- Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.☆55Updated this week
- Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML☆65Updated last year
- A simple semantic search engine for scientific papers.☆28Updated 2 years ago
- Translate HTML using Argos Translate☆57Updated 2 years ago
- Finds linguistic patterns effortlessly☆39Updated 2 years ago
- 🌸 Train floret vectors☆18Updated 2 years ago
- Python based Wikidata framework for easy dataframe extraction☆45Updated 2 years ago
- Small python package to measure OCR quality and other related metrics.☆26Updated last year
- 🐍 Python bidding for the Hora Approximate Nearest Neighbor Search Algorithm library☆73Updated 4 years ago
- Next-generation Punkt sentence boundary detection with zero dependencies☆27Updated 2 months ago
- Sample implementation of natural language image search with OpenAI's CLIP and Elasticsearch or Opensearch.☆73Updated 3 years ago
- Analyze and extract Wikipedia article text and attributes and store them into an ElasticSearch index or to json files (multilingual suppo…☆48Updated 2 years ago
- ☆69Updated 5 years ago