NOISE-RESILIENT DUPLICATE RECORD DETECTION IN MULTI-SOURCE PRODUCT DATABASES USING SIAMESE DEEP NETWORKS AND GRADIENT-BOOSTED MODELS

Title: NOISE-RESILIENT DUPLICATE RECORD DETECTION IN MULTI-SOURCE PRODUCT DATABASES USING SIAMESE DEEP NETWORKS AND GRADIENT-BOOSTED MODELS Keywords - Duplicate Detection; Machine Learning; Deep Learning; Siamese Network; Data Quality; Noisy Databases; Record Linkage. Abstract: Duplicate records are a recurring problem in large product and business databases due to data entry errors, inconsistent formatting, spelling variations and missing fields. These inconsistencies degrade data quality and directly affect analytics, reporting, recommendation systems and downstream machine learning tasks. Traditional rule-based matching approaches perform poorly when the dataset contains moderate noise. This study compares machine learning models (Random Forest and XGBoost) with deep learning models (a feedforward neural network and a Siamese network) for detecting duplicate records in moderately noisy environments. A hybrid dataset combining product and company records from Kaggle, along with synthetically generated noisy duplicates, was used for evaluation. Experimental analysis shows that deep learning methods, especially Siamese networks, outperform traditional ML models in capturing subtle variations in text attributes and inconsistent formatting. The paper concludes with practical guidelines for integrating ML-based duplicate detection in real-world business systems.