HACID: distributed deduplication technology with adaptive layered chunking and efficient indexing

In distributed storage systems, deduplication rate and fingerprint matching efficiency are key challenges. This paper proposes an optimization solution based on hierarchical adaptive chunking and efficient indexing: data chunk fine-grained matching is achieved through hierarchical chunking strategies, and deduplication rate is improved by incorporating multi-dimensional feature similarity calculations; a tree-structured hash layered indexing system is designed to reduce the fingerprint matching scope; and metadata-based pre-filtering rules are applied to reduce invalid matching requests. Experimental results show that, while maintaining system throughput, this solution increases deduplication rate by 25%-30%, reduces fingerprint matching counts by over 60%, and significantly enhances resource utilization and performance in distributed storage systems.