Efficient In-Line Data Deduplication for Non-Volatile Memory Storage Using I/O Causality

I/O causality-based in-line data deduplication (ICID) is a novel approach designed to enhance deduplication efficiency in non-volatile memory (NVM)-enabled storage systems. Traditional chunk-based offline deduplication methods suffer from significant performance overhead due to data chunking and indexing, making them unsuitable for NVM-based storage, which benefits from fine-grained, byte-addressable operations. Unlike conventional in-line deduplication schemes that rely on hash indexes, ICID leverages a B-tree structure to record memory copy operations, enabling causality-based deduplication. To optimize efficiency, ICID introduces two key techniques: grouping memory-copy records targeted to the same page within a B-tree node to enhance data locality and exploiting spatial locality to identify and remove outdated memory-copy records, thereby reducing memory consumption. Evaluations conducted on a system with Intel Optane DC Persistent Memory Modules demonstrate that ICID significantly improves deduplication performance, achieving up to 16× higher deduplication ratios and reducing deduplication time by an average of 47% compared to state-of-the-art methods, particularly benefiting key-value stores like LevelDB.