Identifying Duplicate Customer Records Using Blocking and Supervised Learning Techniques

Identifying duplicate entities in databases is crucial for many organizations and often requires automated processing due to the volume and complexity of records. Blocking techniques can help here by reducing the number of comparisons between records, although they might miss some duplicates based on their configuration. Furthermore, manual review by analysts is still needed to confirm duplicates. To mitigate this problem, we propose an approach that combines blocking techniques with a supervised classification model to simplify duplicate detection. To account for different configurations, we apply hyper-parameter optimization to the entire approach. Our experiments on synthetic and real customer data showed that, despite having an imbalanced dataset, the proposed approach effectively distinguished duplicates from non-duplicates when appropriately trained. These initial results are promising, suggesting potential for a practical implementation in organizational contexts.