A Multimodal Deep Learning Approach for Advancing Liver Disease Diagnosis and Prognosis Prediction

Using a multimodal deep learning framework, we have demonstrated the possibility to predict liver disease subtyping and prognosis with high accuracy. The framework includes over 1600 patients, for which medical imaging and clinical data have been collected. Convolutional Neural Networks (CNNs), Recurrent Neural Network (RNNs), and Long short-term memory (LSTMs) have been combined in the work to extract features from the input data and demonstrate correlation between input and output and model complexity. Such an approach allowed analyzing images and predicting liver diseases based on structural abnormality analysis with the use of CNNs. RNNs and LSTMs, in their turn, allowed combining features for multimodal data source analysis and sequential clinical data analysis providing information about patient history. The obtained results were accurate for each feature extraction and modeling, with the best performance demonstrated by CNN with RNN where the accuracy was 97.8% and another CNN with LSTM where the accuracy was around 94.5%. The model proved to be feasible and accurate in the context of its application to hepatology since multiple data sources were combined in the work. This approach allows accounting for all modes of data source and offers a comprehensive assessment including disease diagnosis and prognosis. The obtained results can be also applied to clinical practice where doctors can add data about their patients to receive more information on diagnosis or prognosis for better decision-making. Finally, the idea to use a multilayered feature from multimodal data sources can be transferable to other types of sequential and structural data and applied to other diseases for their diagnosis and prognosis. However, it is essential to test the framework in other settings to refine it or find limitations to its application.