Joint Coupled Convolutional Dictionary Learning for Multimodal Image Super-Resolution

Multimodal image super-resolution (MISR) aims to enhance the low-resolution (LR) images from the target modality, taking the guidance of high-resolution (HR) images from another modality. We address this problem using translation-invariant convolutional dictionaries that overcome the limitations of traditional patch-based dictionaries. A joint optimization is presented that learns a convolutional dictionary for each modality, i.e., LR images of target modality and HR images of guidance modality with the respective sparse coefficients constrained to reconstruct the HR images of target modality, using two coupling convolutional dictionaries that model the cross-modal dependencies between the two modalities. The effectiveness of the proposed joint coupled convolutional dictionary learning method for MISR is demonstrated using RGB-near infrared (RGB-NIR) and RGB-multispectral (RGB-MS) datasets. Experimental results show improved reconstruction performance of the proposed method, both in peak signal to noise ratio (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;\!\!1$</tex-math></inline-formula> dB) and structural similarity (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;\!\!1\%$</tex-math></inline-formula>) on most images compared to benchmark methods, even with limited training data.