Research Progress of Encoder–Decoder Network in Image Fusion

A B S T R A C TImage fusion is a research hotspot in the field of computer vision, which obtains complementary information from the different modal images and then presents it in a comprehensive, highly information-cohesive fused image. Encoder-decoder networks are an important technical method in the field of image fusion. This paper reviewed the research status and development of the Encoder-decoder networks in the field of image fusion from the following aspects: Firstly, the18 image fusion-related datasets and the 22 open-source codes of deep learning-based image fusion are summarized. Secondly, the model structure of encoder-decoder networks and the basic principles for image fusion based on the encoder-decoder network are described. Thirdly, encoder-decoder networks fusion models are summarized into 4 aspects: Convolutional Auto-encoder network(CAE Net), Sparse Auto-encoder network(SAE Net), U-shape Encoder-decoder network(U-net) and Variational Auto-encoder network(VAE Net),in U-net, there are 5 types: U-net incorporating residual connection mechanism (Res U-net), U-net incorporating dense connection mechanism (Dens U-net), U-net incorporating multiscale connection mechanism (MS U-net), U-net incorporating attention mechanism (Att U-net) and U-shaped Encoder-decoder networks(U-net) combined with Generative Adversarial Network(GAN U-net). Fourthly, the existing image fusion evaluation metrics are summarized from both subjective and objective aspects. Fifthly, the application of Encoder-decoder network in 5 kinds of image fusion, such as multi-exposure image fusion, multi-focus image fusion, infrared and visible image fusion, medical image fusion and remote sensing image fusion are summarized. Finally, there is a discussion of the main challenges faced by encoder-decoder networks in the image fusion domain and an outlook on future directions. This paper systematically analyzed the application of encoder-decoder networks in the image fusion areas, which has positive implications for the progress of image fusion technology.