Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
☆18Aug 28, 2023Updated 2 years ago
Alternatives and similar repositories for deduplication
Users that are interested in deduplication are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Find duplicate text files.☆14Jan 14, 2025Updated last year
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- ☆14Dec 9, 2021Updated 4 years ago
- Crawling some e-commerce site in Indonesia (blibli, bukalapak, lazada, mataharimall, and tokopedia) using python scrapy and save the craw…☆10Jan 28, 2017Updated 9 years ago
- 📺 Self-hosted, open source YouTube subscription management system☆11Apr 2, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- level2-nlp-generationfornlp-nlp-05-lv3 created by GitHub Classroom☆14Jan 5, 2025Updated last year
- 나무위키, 위키피디아, 다음블로그, 티스토리, 유튜브, 네이트판 크롤러☆13Feb 20, 2026Updated 5 months ago
- ☆15Aug 25, 2025Updated 10 months ago
- TensorRT☆11Sep 22, 2020Updated 5 years ago
- PostgreSQL database and a corresponding Flask web interface to store music library information dynamically with multiple linked versions …☆10Jun 2, 2023Updated 3 years ago
- Hydrus Network Client API application that makes it easier to keep your Hydrus tags in shape☆14Jul 28, 2021Updated 4 years ago
- a rust library to find near-duplicate video files☆25Apr 23, 2025Updated last year
- RAG-Fusion implementation using Langchain, Weaviate and OpenAI☆13Oct 31, 2023Updated 2 years ago
- Supercharged pandas indexing☆11Mar 28, 2021Updated 5 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Demo of obsidiantools Python package (for Binder)☆13Jul 8, 2025Updated last year
- ☆14Apr 13, 2026Updated 3 months ago
- Repo for the IDESSAI 2024 course on modeling audio with discrete tokens.☆13Sep 13, 2024Updated last year
- Lets you reorganize videos in your watch later list to various playlists (for watch later hoarders like me!)☆10Feb 12, 2015Updated 11 years ago
- The program can be used to scrape the content from an article from web by an input of a set of URLs in a text file or a URL. This project…☆18Aug 5, 2020Updated 5 years ago
- Cache any function call's to Deta base.☆13Oct 9, 2022Updated 3 years ago
- ☆18Jun 10, 2026Updated last month
- Collection of small scripts to generate update feeds☆12Mar 9, 2023Updated 3 years ago
- Telegram > OpenAI > Read Later [instapaper/pocket/omnivore]☆16Jul 15, 2023Updated 3 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆18Jul 4, 2017Updated 9 years ago
- Crawl APP data from Google Play Store in Indonesia☆24Jul 23, 2018Updated 7 years ago
- ☆12Mar 28, 2025Updated last year
- ☆11Apr 24, 2023Updated 3 years ago
- The most powerful and fastest YouTube searching Python library.☆13Oct 25, 2022Updated 3 years ago
- A chatbot made using the Chatterbot library in Python and locally hosted using Streamlit. Dataset used were collected during ConvAI2 comp…☆16Jul 26, 2021Updated 4 years ago
- TUI (Text User Interface) - Get Instant feedback for your sh commands. Explore and play with your queries 🚀☆12Aug 30, 2025Updated 10 months ago
- I have created a dataset of Image-Text-Pairs by using the cosine similarity of the CLIP embeddings of the image & it's caption derrived f…☆17Apr 22, 2021Updated 5 years ago
- ~/.dotfiles: Configuration for various applications☆14Jul 31, 2025Updated 11 months ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Web Scraping Service☆15May 11, 2026Updated 2 months ago
- Repo used to share the demos from the "Seeking SRE" (O'Reilly Media, 2018) used in D.C. Vienna 2017☆14Sep 27, 2017Updated 8 years ago
- ☆39Jul 25, 2024Updated last year
- PPO with Hindsight Experience Replay (HER)☆12May 8, 2018Updated 8 years ago
- My dotfiles☆15May 7, 2026Updated 2 months ago
- Knowledge extraction from semi-structured web.☆13Mar 25, 2024Updated 2 years ago
- 大规模中文语料☆44Nov 5, 2019Updated 6 years ago