The Standard Deviation Score: a novel similarity metric for data analysis

Abstract The ability to measure similarity or distance between data points is critical for various analytical tasks, including classification, clustering, and anomaly detection. However, traditional distance metrics such as Euclidean, Manhattan, and Hamming often struggle with mixed data types, varying attribute scales, and noise, limiting their robustness in diverse datasets. This paper introduces the Standard Deviation Score (SD-score), a novel similarity metric designed to address these challenges. By transforming traditional distance values into standard deviation units relative to a target point, the SD-score enables robust and interpretable similarity assessments. Extensive experimental evaluations demonstrate that the SD-score consistently outperforms conventional metrics in accuracy, precision, recall, and F-score within the k-Nearest Neighbors classification framework. Also, a comprehensive evaluation of the SD-score’s performance across Gaussian, skewed, and multimodal distributions showed promising results in the cluster coherence experiment, in which the Silhouette score was measured through the K-means clustering algorithm, emphasizing its adaptability to real-world data complexities. Additionally, the experiments detail improved handling of mixed numerical, ordinal, and categorical data types through a unified framework. The proposed metric incorporates inherent normalization mechanisms, reducing sensitivity to outliers and ensuring consistency across varying data scales and distributions, making it a versatile tool for real-world applications. This advancement in similarity measurement paves the way for more accurate and efficient data analysis across multiple domains.