Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆19Aug 28, 2023Updated 3 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Price options by fitting a Lévy distribution☆10Jan 20, 2021Updated 5 years ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- level2-nlp-generationfornlp-nlp-05-lv3 created by GitHub Classroom☆14Jan 5, 2025Updated last year
- data related codebase for polyglot project☆19Mar 30, 2023Updated 3 years ago
- Multiple ways of chunking for data deduplication: Fixed size chunking, Content defined chunking, and File based chunking.☆19Dec 20, 2013Updated 12 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- 나무위키, 위키피디아, 다음블로그, 티스토리, 유튜브, 네이트판 크롤러☆13Feb 20, 2026Updated 6 months ago
- ☆15Aug 25, 2025Updated last year
- Functional composable pipelines allowing clean separation of the business logic and its implementation☆11Sep 6, 2025Updated 11 months ago
- Repository provides an implementation of D*Lite algorithm adapted for single-shot grid-based 2D environment.☆22Oct 26, 2017Updated 8 years ago
- A python library / model for creating co-references between AMR graph nodes.☆11Dec 11, 2022Updated 3 years ago
- ☆10Jun 22, 2020Updated 6 years ago
- PostgreSQL database and a corresponding Flask web interface to store music library information dynamically with multiple linked versions …☆10Jun 2, 2023Updated 3 years ago
- RAG-Fusion implementation using Langchain, Weaviate and OpenAI☆13Oct 31, 2023Updated 2 years ago
- Lets you reorganize videos in your watch later list to various playlists (for watch later hoarders like me!)☆10Feb 12, 2015Updated 11 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- IEEE-CIS Fraud Detection Kaggle Competition Code☆10Oct 3, 2019Updated 6 years ago
- The program can be used to scrape the content from an article from web by an input of a set of URLs in a text file or a URL. This project…☆18Aug 5, 2020Updated 6 years ago
- Cache any function call's to Deta base.☆13Oct 9, 2022Updated 3 years ago
- Sample solution to build a deployment pipeline for Amazon SageMaker.☆14Jul 18, 2022Updated 4 years ago
- Telegram > OpenAI > Read Later [instapaper/pocket/omnivore]☆16Jul 15, 2023Updated 3 years ago
- FUSE file system for browsing tags from photo management applications as a file system☆12Jan 2, 2018Updated 8 years ago
- A chatbot made using the Chatterbot library in Python and locally hosted using Streamlit. Dataset used were collected during ConvAI2 comp…☆16Jul 26, 2021Updated 5 years ago
- 实时监听文件夹变动并同步到ftp服务端☆13Jul 5, 2019Updated 7 years ago
- Implementing the OPRO paper☆17Sep 18, 2023Updated 2 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- MyAssistant Playground --powered by Bedrock Claude & AutoGen☆12Mar 26, 2024Updated 2 years ago
- ~/.dotfiles: Configuration for various applications☆14Jul 31, 2025Updated last year
- Repo used to share the demos from the "Seeking SRE" (O'Reilly Media, 2018) used in D.C. Vienna 2017☆14Sep 27, 2017Updated 8 years ago
- PPO with Hindsight Experience Replay (HER)☆12May 8, 2018Updated 8 years ago
- This project implenments the OSPF using Dijkstra algorithm (Open Shortest Path First) network protocol in python. Link-State Routing pr…☆12Sep 1, 2017Updated 8 years ago
- Spreadsheet Power Tools GUI - Pandas, PySide, SQLAlchemy☆13Nov 19, 2017Updated 8 years ago
- A nice way to display CLI Github user / repo / organization info inspired in neofetch☆16Dec 1, 2024Updated last year
- Build wordlists from the common-crawl index☆11Oct 9, 2022Updated 3 years ago
- 🐎 A fast implementation of the Aho-Corasick algorithm using the compact double-array data structure. (Python wrapper for daachorse)☆21Aug 12, 2026Updated 2 weeks ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- ☆14Sep 10, 2021Updated 4 years ago
- AWS Quick Start Team☆14Oct 3, 2024Updated last year
- Chatbot using Tensorflow (Model is transformer) ko☆30Dec 10, 2018Updated 7 years ago
- A collection of NLP resources for Malay☆26Sep 18, 2018Updated 7 years ago
- An open dataset of all chat activity in Free Code Camp's Gitter.im chatrooms☆17Dec 24, 2016Updated 9 years ago
- Takes a list of vertices and faces, giving you back an array of individual triangles.☆12Nov 18, 2015Updated 10 years ago
- Automated Safety Testing of Large Language Models☆17Jan 31, 2025Updated last year