Use of Attention Maps to Enrich Discriminability in Deep Learning Prediction Models Using Longitudinal Data from Electronic Health Records

Background: In predictive modelling, particularly in fields such as healthcare, the importance of understanding the model’s behaviour rivals, if not surpasses, that of discriminability. To this end, attention mechanisms have been included in deep learning models for years. However, when comparing different models, the one with the best discriminability is usually chosen without considering the clinical plausibility of their predictions. Objective: In this work several attention-based deep learning architectures with increasing degrees of complexity were designed and compared aiming to study the balance between discriminability and plausibility with architecture complexity when working with longitudinal data from Electronic Health Records (EHRs). Methods: We developed four deep learning-based architectures with attention mechanisms that were progressively more complex to handle longitudinal data from EHRs. We evaluated their discriminability and resulting attention maps and compared them amongst architectures and different input processing approaches. We trained them on 10 years of data from EHRs from Catalonia (Spain) and evaluated them using a 5-fold cross-validation to predict 1-year all-cause mortality in a subsample of 500,000 people over 65 years of age. Results: Generally, the simplest architectures led to the best overall discriminability, slightly decreasing with complexity by up to 8.7%. However, the attention maps resulting from the simpler architectures were less informative and less clinically plausible compared to those from more complex architectures. Moreover, the latter could give attention weights both in the time and feature domains. Conclusions: Our results suggest that discriminability and more informative and clinically plausible attention maps do not always go together. Given the preferences within the healthcare field for enhanced explainability, establishing a balance with discriminability is imperative.