Deep Learning-Based Early Detection of Rare Diseases Using Electronic Health Records

Early detection of rare diseases is one of the ongoing challenges in clinical practice because their prevalence is low, presentations are heterogeneous, and their diagnoses are complex. In this work, we introduce a new deep learning approach based on a Hierarchical Temporal Transformer (HTT) to detect rare diseases from clinical multi-dimensional patient data stored in electronic health records (EHR). Our model is tailored to recognize complex patterns in patient data over time and overcome extreme class imbalance by introducing a focal loss function. We perform an extensive comparison with conventional machine learning and deep learning baselines on three big-scale real-world EHR datasets covering over 170,000 patients with multiple rare diseases like Gaucher, ALS, and AADC deficiency. Our model outperforms baselines with a significant margin with an F1-score of 0.69, AUC of 0.89, and detects disease up to 12 days in advance of clinical detection. Our model models generalize well across datasets from other institutions and shows stable, balanced performance across demographic sub-populations. Besides predictive performance, the model provides clinical interpretability in the form of its temporal attention mechanism to flag medical relevant features (e.g., splenomegaly, thrombocytopenia) that are known to be associated with disease. An exhaustive ablation study also ensures the contribution of the key architectural elements such as hierarchical embedding and positional encodings. Our work shows the potential of deep learning in diagnosing rare diseases early and indicates the clinical utility of interpretable AI in clinical care in real-world healthcare. Our proposed model is a transferable and scalable solution that can facilitate early intervention and reduce the delay in diagnostics and improve the treatment of people with rare diseases