Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆19Aug 28, 2023Updated 2 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆39Jul 28, 2023Updated 3 years ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- ☆14Dec 9, 2021Updated 4 years ago
- level2-nlp-generationfornlp-nlp-05-lv3 created by GitHub Classroom☆14Jan 5, 2025Updated last year
- data related codebase for polyglot project☆19Mar 30, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Beyond LM: How can language model go forward in the future?☆15Apr 30, 2023Updated 3 years ago
- A GoLang implementation of Fiche as provided by solusipse/fiche☆11Jan 5, 2018Updated 8 years ago
- Functional composable pipelines allowing clean separation of the business logic and its implementation☆11Sep 6, 2025Updated 11 months ago
- Copilot with deepseek and more...☆13Mar 7, 2025Updated last year
- super-Django-CC is a simle web interface for commoncrawl.org☆14Dec 8, 2022Updated 3 years ago
- ☆10Jun 22, 2020Updated 6 years ago
- Rust bindings to the Knot Resolver library (also known as libkres)☆18Apr 2, 2019Updated 7 years ago
- a rust library to find near-duplicate video files☆25Apr 23, 2025Updated last year
- Python package for deduplication/entity resolution using active learning☆82Aug 24, 2024Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Fast, lightweight MaxMind GeoIP lookup server written in Rust☆18Jul 26, 2026Updated 2 weeks ago
- Demo of obsidiantools Python package (for Binder)☆13Jul 8, 2025Updated last year
- ☆16Nov 19, 2023Updated 2 years ago
- ☆14Apr 13, 2026Updated 3 months ago
- A type decoder for objective c types☆14Oct 20, 2024Updated last year
- IEEE-CIS Fraud Detection Kaggle Competition Code☆10Oct 3, 2019Updated 6 years ago
- The program can be used to scrape the content from an article from web by an input of a set of URLs in a text file or a URL. This project…☆18Aug 5, 2020Updated 6 years ago
- Cache any function call's to Deta base.☆13Oct 9, 2022Updated 3 years ago
- Collection of small scripts to generate update feeds☆12Mar 9, 2023Updated 3 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Upgrade your checkpost security with AUTOMATIC NUMBER PLATE RECOGNITION system.☆11May 1, 2025Updated last year
- ☆12Mar 28, 2025Updated last year
- A chatbot made using the Chatterbot library in Python and locally hosted using Streamlit. Dataset used were collected during ConvAI2 comp…☆16Jul 26, 2021Updated 5 years ago
- Code for NeurIPS2021 submission "A Surrogate Objective Framework for Prediction+Programming with Soft Constraints"☆13Aug 30, 2021Updated 4 years ago
- Implementing the OPRO paper☆16Sep 18, 2023Updated 2 years ago
- MyAssistant Playground --powered by Bedrock Claude & AutoGen☆12Mar 26, 2024Updated 2 years ago
- Variationist: Exploring Multifaceted Variation and Bias in Written Language Data (ACL 2024 demo track)☆10Jan 31, 2026Updated 6 months ago
- Repo used to share the demos from the "Seeking SRE" (O'Reilly Media, 2018) used in D.C. Vienna 2017☆14Sep 27, 2017Updated 8 years ago
- Web Scraping Service☆15May 11, 2026Updated 2 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ☆39Jul 25, 2024Updated 2 years ago
- PPO with Hindsight Experience Replay (HER)☆12May 8, 2018Updated 8 years ago
- My dotfiles☆15May 7, 2026Updated 3 months ago
- This project implenments the OSPF using Dijkstra algorithm (Open Shortest Path First) network protocol in python. Link-State Routing pr…☆12Sep 1, 2017Updated 8 years ago
- Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network☆21Dec 20, 2017Updated 8 years ago
- A nice way to display CLI Github user / repo / organization info inspired in neofetch☆15Dec 1, 2024Updated last year
- 𝗬𝗢𝗨𝗧𝗨𝗕𝗘 𝗜𝗣 𝗕𝗔𝗡 𝗜𝗦𝗦𝗨𝗘 𝗦𝗢𝗟𝗩𝗘𝗗. 𝗠𝗨𝗦𝗜𝗖 𝗕𝗢𝗧 𝗡𝗢 🌱𝗟𝗔𝗚 𝗙𝗔𝗦𝗧 𝗦𝗣𝗘𝗘𝗗 (V2)🏵️𝗕𝗢𝗧 ʏᴛ-ᴅʟᴘ ᴇʀʀᴏʀ …☆10Jul 1, 2026Updated last month