文档去重功能是为了解决搜索引擎的文档语义重复的问题，方法是多重哈希下的语义指纹算法。
☆12Aug 17, 2013Updated 13 years ago
Alternatives and similar repositories for deduplication-detecting
Users that are interested in deduplication-detecting are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- a deduplication tool☆17Jul 27, 2019Updated 7 years ago
- Python脚本实现千万级文本数据快速去重☆19Mar 14, 2016Updated 10 years ago
- file deduplication in go☆11Dec 20, 2022Updated 3 years ago
- A Python tool to search for and remove duplicated files in messy datasets☆15Dec 23, 2024Updated last year
- A merged read deduplication tool capable to perform merged read deduplication on single end data.☆16Sep 4, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Deduplication for cfDNA sequencing data☆11Jul 5, 2017Updated 9 years ago
- Get a list of deduped files on a ZFS filesystem☆14Oct 14, 2020Updated 6 years ago
- 基于互信息和邻接信息熵的中文分词和新词发现☆14Jan 22, 2019Updated 7 years ago
- String deduplication package for Go☆19Jan 10, 2024Updated 2 years ago
- Find near-duplicate documents using minhashing implemented in Go.☆16Dec 22, 2015Updated 10 years ago
- Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.☆19Aug 28, 2023Updated 3 years ago
- 基于gensim模块,训练LDA(Latent Dirichlet Allocation)模型,用于计算长短文本的相似度.☆12Nov 25, 2020Updated 5 years ago
- A file system that can be used to compare different deduplication algorithms.☆17Jul 15, 2026Updated 3 months ago
- Find duplicate text files.☆14Jan 14, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆18Apr 3, 2023Updated 3 years ago
- 文本去重算法，研究自推荐系统中新闻的去重，采用了雅虎的Near-duplicates and shingling算法，服务端用c实现，客户端用java实现，利用thrift框架进行通信，为了提高扩展性，去重可以在服务端实现，服务器也提供了计算的接口，方便客户端自己扩展☆24Feb 25, 2014Updated 12 years ago
- Python MaxMind DB writer☆14Apr 3, 2019Updated 7 years ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- default visualizations that come packaged with the lightning viz notebook☆12Apr 18, 2016Updated 10 years ago
- RapidCDC: Leveraging Duplicate Locality to Accelerate Chunking in CDC-based Deduplication Systems☆17May 25, 2020Updated 6 years ago
- Codex skill for generating and auditing mature SaaS and data admin systems☆20Aug 4, 2026Updated 3 months ago
- Python library and dashboard for hyperparameter search and model training for computer vision tasks based on PyTorch, Optuna, FiftyOne, D…☆17Jul 14, 2023Updated 3 years ago
- ✨ Epris is a JavaScript library that simplifies interface development☆26May 30, 2022Updated 4 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Express4+angularjs1.4+bootstrap3+mysql+nodejs☆20Sep 11, 2018Updated 8 years ago
- Rabin hashing and content-defined chunking for Go☆21Sep 11, 2017Updated 9 years ago
- Performs a poor man's file deduplication recursively on a directory. Deletes duplicate files, and creates symbolic links in their place.☆31Dec 2, 2011Updated 14 years ago
- Implementation of DSGAN (not fully completed)☆13Dec 28, 2019Updated 6 years ago
- 🕹️ Group and deduplicate concurrent tasks☆32Updated this week
- A Go library implementing a buzhash rolling hash function☆30Aug 16, 2016Updated 10 years ago
- The pytorch implementation of relational extraction models with PCNN feature extractor and multi-instance learning☆16Mar 8, 2018Updated 8 years ago
- ☆21May 24, 2016Updated 10 years ago
- Multiple ways of chunking for data deduplication: Fixed size chunking, Content defined chunking, and File based chunking.☆19Dec 20, 2013Updated 12 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- ☆18Jun 27, 2016Updated 10 years ago
- webexe pc端windows demo（Qt版），基于移动直播，通过RoomService管理业务后台（参考Demo：roomservice_server_java），用于与小程序<live-room>、<rtc-room>标签互通。☆13Sep 29, 2018Updated 8 years ago
- Fast duplicate file detection library☆27Jan 5, 2017Updated 9 years ago
- Elasticsearch Shards Allocation visualization as a kibana plugin☆26May 18, 2017Updated 9 years ago
- 数据虫巢(微信号blogchong)公众号技术文章合集。虫巢出品，不说优品，最起码也得算个良品呐~~☆25Feb 13, 2017Updated 9 years ago
- A Python FUSE file system that features transparent deduplication and compression which make it ideal for archiving backups.☆141Jul 22, 2010Updated 16 years ago
- 分享一些技术类的但不仅仅是技术类的pdf书籍供大家学习☆10Jul 31, 2018Updated 8 years ago