Consistently Evaluating Record Linkage Classifiers

Record linkage is the process of identifying records that refer to the same real-world entity within or across databases. If training data in the form of true matches (two records referring to the same entity) and true non-matches (two records referring to different entities) are available, then record linkage can be viewed as a supervised classification problem. Performance measures such as precision, recall, and the F-measure, are commonly used to evaluate the linkage quality obtained with a trained classifier. However, as we show in this article, comparing multiple classifiers using such measures can lead to inconsistent evaluation because for a given measure the same result can be obtained from different classification outcomes. This can cause a suboptimal classifier being selected, which can result in linked data sets of poor quality and possibly wrong decisions being made. To overcome this problem, we propose the Consistent Record Linkage (CRL) measure, an application focused evaluation method that ensures record linkage classifiers are assessed in a fair and transparent way. The CRL-measure allows a user to define the maximum error rates that are acceptable for their linkage application, and it provides practically useful information about the robustness of a classifier with respect to the range of classification thresholds obtained with these error rates. We evaluate the CRL-measure on both synthetic and real-world data sets using multiple classifiers to show its advantage over standard performance measures.