GENERATIVE ADVERSARIAL NETWORKS (GANs) FOR OPTIMIZING DATA DEDUPLICATION IN LARGE DATASETS

Abstract This study explores the application of Generative Adversarial Networks (GANs) to the challenge of data deduplication in large datasets. Traditional deduplication methods, often reliant on hashing and exact matching, suffer from limitations in scalability and effectiveness when encountering near-duplicates. We propose a novel GAN-based framework that recasts deduplication as an adversarial learning problem, where a generator creates synthetic duplicates and a discriminator learns to identify them. By transforming deduplication into a predictive modeling task, the approach achieves improved detection accuracy and a substantial reduction in time complexity from O(n) to approximately O(log n). Empirical evaluations on synthetic large-scale datasets demonstrate significant gains in precision, recall, and runtime efficiency compared to conventional methods. This approach offers a scalable, intelligent alternative for managing redundant data in modern storage systems.