Marcnuth / deduplicationLinks
Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆18Updated 2 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below
Sorting:
- ☆14Updated last year
- Indri search implementation on top of Lucene search engine☆35Updated last year
- An open-source NLP library: fast text cleaning and preprocessing☆23Updated 3 years ago
- This library builds a graph-representation of the content of PDFs. The graph is then clustered, resulting page segments are classified an…☆22Updated 5 years ago
- FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction☆24Updated 3 years ago
- 📑 Python Package to reconstruct the original continuous text from PDFs with language models☆32Updated 2 years ago
- ☆69Updated 4 years ago
- This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…☆33Updated 2 years ago
- An example of graph embeddings for wikipedia page recommendations☆11Updated 4 years ago
- Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML☆64Updated 9 months ago
- LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development☆19Updated 2 years ago
- Analyze and extract Wikipedia article text and attributes and store them into an ElasticSearch index or to json files (multilingual suppo…☆47Updated 2 years ago
- A cost estimator for OpenAI API calls in tqdm loops.☆20Updated 11 months ago
- Complex data extraction and orchestration framework designed for processing unstructured documents. It integrates AI-powered document pip…☆73Updated this week
- 🚂 Fine-tune OpenAI models for text classification, question answering, and more☆17Updated 2 years ago
- ☆91Updated 3 years ago
- Fast fuzzy text search☆11Updated 2 years ago
- Integration between Reaction ECommerce and Accelerated Text to provide product descriptions for an e-shop.☆12Updated 4 years ago
- Documentation effort for the BookCorpus dataset☆34Updated 4 years ago
- Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.☆45Updated last year
- Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.☆62Updated this week
- Scripts to parse arxiv documents for NLP tasks☆18Updated 2 years ago
- Small python package to measure OCR quality and other related metrics.☆25Updated last year
- SMASHED is a toolkit designed to apply transformations to samples in datasets, such as fields extraction, tokenization, prompting, batchi…☆35Updated last year
- Neural Elastic Inference and Search☆19Updated 5 years ago
- Translate HTML using Argos Translate☆53Updated 2 years ago
- Interpretable feature construction from taxonomies for text classification☆18Updated 3 years ago
- Rust python bindings for symspell☆21Updated last year
- Open source library for few shot NLP☆78Updated 2 years ago
- Sentence Embedding as a Service☆15Updated 4 months ago