Explainable variable-weight multi-modal based deep learning framework for catheter malposition detection

Hospital patients may have catheters and lines inserted for quick administration of medicines or medical tests. However, a misplaced catheter can cause serious complications, even death. Recently, deep learning frameworks have shown their potential to assist in detecting catheter malposition in radiography. However, the deep learning malposition detection frameworks meet three main challenges: (1) Most approaches rely heavily on visual information, requiring models with many parameters for accurate detection. (2) Geometric information in radiography that is important for experts for decision making is often underutilized due to the inherent complexities in accurately extracting and integrating it with visual information. (3) Feature significance in catheter status detection is often underexplored, making the framework difficult to interpret and requiring a mechanism to highlight key factors influencing decisions. Therefore, to address these challenges, an explainable variable-weight multimodal based deep learning framework is proposed to fuse the visual and geometric information in the radiography for catheter malposition detection. The convolution neural network (CNN) stream and the graph convolution neural network (GCN) stream, with few learnable parameters, are designed to extract the visual and geometric information without compromising performance. The cross-modal attention block is proposed to capture the relationship between visual and geometric information. Furthermore, the multimodal variable-weight structure is proposed to fuse different modalities based on their significance. To visualize the contribution of each modality, the multimodal class activation map (MCAM) is designed to visualize the activated region in radiography, showing where the framework focuses. The proposed method obtains state-of-the-art performance, gaining 0.8816 mean AUC with 7.62 million parameters. • A unified model integrating visual and geometric information is proposed. • The significance of each modality in decision-making is verified. • The proposed method can gain state-of-the-art performance with fewer parameters. • The region where the framework focuses after the multimodal fusion is visualized. • The method assists clinicians in interpreting the machine decision process.