Optimizing Deduplication Parameters via a Change-Estimation Analytical Model

Variable-sized, content-defined deduplication is a technique to find and eliminate redundant chunks of data for efficient data backups, reduced data transfers, and reduced data-storage overheads. For big datasets, especially with incremental updates over time such as backups and gathered data, deduplication makes data management faster and more efficient. While many existing deduplication systems use default expected chunk lengths such as 4 KB or 8 KB, they are suboptimal. Poorly optimized deduplication systems can significantly increase storage costs and network usage, making large datasets prohibitively expensive to manage. We present the design, implementation, and an empirical validation of our Deduplication Change-Estimation Analytical Model (DCAM) which predicts the performance of sliding window-based deduplication parameters on any given dataset, to be used for parameter optimization. Our empirical evaluation includes workloads based on source code (Linux kernel, Kubernetes, TensorFlow), open-research datasets (CORD-19), and articles (Wikipedia). Validated using both our system and the Destor deduplication system, a DCAM-based search finds deduplication parameters that require up to 3.8× less storage relative to a common baseline. DCAM Search optimizes parameters up to 19.8× faster than previously possible, and the size of the resulting deduplicated datasets are all within 5.15% of the best results found by searching using actual deduplication.