Novel Perspective on Ensemble Clustering for Persistent Cluster Patterns: A Case Study in Disease Cluster Discovery

Clustering is the process of finding natural groups within a data set such that patterns within a group are more similar than patterns belonging to different groups. It has been used in a wide range of scientific and engineering disciplines. Yet, clustering is also a difficult unsupervised problem without an absolute ground truth. In practice, the "best" clustering method is the one that produces the most interpretable results. There is no universal optimum way to select the number of clusters. Our ultimate goal is to enable domain experts in high-stakes fields to make informed decisions when choosing a final, insightful, and actionable clustering solution for complex problems. In such challenging scenarios, ensemble clustering is a technique where multiple clustering results are combined to produce a more robust and stable final clustering. Here, inspired by a medical application, we take a novel approach to ensemble clustering; rather than focusing on an optimum global clustering, we identify persistent cluster(s) by multiple techniques across multiple clustering experiments among patients from Rush University Medical Center in Chicago. The key novelty resides in reducing the uncertainty in clustering, in the absence of ground truth, by looking at persistent clusters across the ensemble clustering. In healthcare, physicians aim to treat a patient based on a series of symptoms that contribute to a single disease. Because most clinical guidelines focus on individual diseases, managing patients with multimorbidity—when a person has multiple (chronic) conditions—can be challenging. We aim to discover clusters of diseases with some level of confidence to assist physicians in diagnosing and treating patients with multimorbidity. In this case study, we identify and thoroughly characterize a cluster of diseases that affects primarily women and persists through clustering the data from k=16 to k=20 clusters using three clustering techniques. Additionally, our first proposed multimorbidity cluster identified a network connecting asthma, breast cancer, hip/pelvic fracture, endometrial cancer, and non-Alzheimer’s dementia. We use multiple clustering techniques, consensus metrics, as well as graph analysis and visualization to provide confidence and foster trust from domain experts in our results. We are working with physicians to provide a medical explanation for this data-driven disease multimorbidity discovery.