Cardiac MR Image Semantic Segmentation based on Joint Dual-stream CNNs

Image semantic segmentation is the pixel-level understanding and classification of visual regions of interest. In this paper, a novel cardiac magnetic resonance image semantic segmentation architecture is presented based on a joint dual-stream convolutional neural network (CNN) called JDSCNN. The method effectively integrates multi-level, multi-layer and multi-scale context and detailed information to perform semantic segmentation. First, a semantic stream and a detail stream are used in the proposed network to extract the context and edge features respectively. Specifically, for the semantic stream, a gated fully fusion module is integrated as the skip connection to extract multi-level features. Besides, a feature pyramid module is proposed to concatenate multi-layer features in the decoder path. A feature aggregation module is also used to obtain multi-scale context. For the detail stream, a gated shape CNN exploiting multi-level context is proposed to extract edge features of the objects. Second, we integrate an inter-stream fusion module to fuse and propagate the features in each stream bidirectionally. Finally, a multi-task loss function combining the edge information and segmentation information is applied. JDSCNN is evaluated on the public cardiac image dataset, which achieves significant results for the left ventricle, right ventricle and myocardium compared with the state-of-the-art algorithm. Our code is available at: https://github.com/csathu/JDSCNN.