Marcnuth / deduplication
Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆16Updated last year
Related projects ⓘ
Alternatives and complementary repositories for deduplication
- A curated list of awesome open source tools and commercial products to catalog, version, and manage data 🚀☆25Updated 2 years ago
- An open-source NLP library: fast text cleaning and preprocessing☆23Updated 3 years ago
- Neural Solr = Solr 9 + Mighty Inference + Node☆16Updated 2 years ago
- Datasette plugin for searching all searchable tables at once☆19Updated 2 months ago
- A file utility for accessing both local and remote files through a unified interface.☆35Updated 3 months ago
- Scripts supporting the development and serving the Roots Search Tool - https://hf.co/spaces/bigscience-data/roots-search☆10Updated last year
- Neural Elastic Inference and Search☆19Updated 4 years ago
- Code examples for Google Natural Language API.☆13Updated 5 years ago
- Functional composable pipelines allowing clean separation of the business logic and its implementation☆11Updated 5 months ago
- Generate random date(time) in Python.☆10Updated 8 months ago
- code and data used to build a training dataset for dragnet models☆10Updated 3 years ago
- Transform Oracle PL/SQL Code to Python☆11Updated 11 years ago
- Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.☆51Updated last week
- CLI based diff viewer☆24Updated 3 years ago
- Repository to allow collaboration between Cycle Labs Cloud community in support of the community.☆9Updated 2 years ago
- Interpretable feature construction from taxonomies for text classification☆18Updated 2 years ago
- Custom Python functions for working with SQLite FTS4☆22Updated 2 years ago
- A utility for labeling clusters of text data.☆28Updated 3 years ago
- gRPC server for hnswlib☆14Updated last year
- A simple library for training named entity recognition model from partially annotated data☆21Updated 11 months ago
- 🚂 Fine-tune OpenAI models for text classification, question answering, and more☆16Updated last year
- Datamallet is a python library which contains several helper functions and module for the common tasks in a typical data science workflow…☆11Updated 2 years ago
- Loadable spellfix1 extension for sqlite as python package☆25Updated 6 months ago
- My dot files in one place - extensively edited over time. Your mileage may vary☆2Updated 8 years ago
- Go library that provides easy-to-use interfaces and tools for TensorFlow users, in particular allowing to train existing TF models on .ta…☆14Updated 7 months ago
- Various Jupyter notebooks about Common Crawl data☆46Updated 2 years ago
- Train a model, and detect gibberish strings with it.☆59Updated 2 years ago
- Python bindings for the Google's FarmHash☆37Updated 2 months ago
- Cortex-compatible model server for Python and TensorFlow☆16Updated last year
- Datasette enrichment for analyzing row data using OpenAI's GPT models☆19Updated 5 months ago