PhaseVSRnet: Deep complex network for phase-based satellite video super-resolution

Satellite video super-resolution (SR) aims to generate high-resolution (HR) frames from multiple low-resolution (LR) frames. To exploit motion cues under complicated motion patterns, most CNN-based methods first perform motion compensation and then aggregate motion cues in aligned frames (features). However, due to the low spatial resolution of satellite videos, the moving scales are usually subtle and difficult to be captured in the spatial domain. Furthermore, various scales of moving objects challenge current satellite video SR methods in motion estimation and compensation. To address these challenges for satellite video SR, we propose PhaseVSRnet to convert satellite video frames into the phase domain. By representing the motion information with phase shifts, the subtle motions are enlarged in the phase domain. Specifically, our PhaseVSRnet employs deep complex convolutions to better exploit the inherent correlation of complex-valued decompositions obtained by complex-valued steerable pyramids. Then, we adopt a coarse-to-fine motion compensation mechanism to eliminate phase ambiguity at different levels. Finally, in hierarchical reconstruction stage, we use the multi-scale fusion module to aggregate features from multiple levels and use an upsampling layer to upsample the feature maps for resolution enhancement. With PhaseVSRnet, we effectively address the subtle motions and varying scales of moving objects in satellite videos. We assess its performance on a satellite video SR dataset from Jilin-1 satellites and evaluate its generalization ability on another SR dataset from OVS-1 satellites. The results show that PhaseVSRnet effectively captures motion cues in the phase domain and exhibits strong generalization capability across different satellite sensors in unseen scenarios. • The first end-to-end phase-based network for satellite video super resolution. • Treating phase and amplitude together for harnessing their inherent correlation. • A coarse-to-fine motion compensation mechanism to eliminate phase ambiguity. • A multi-scale fusion strategy to aggregate features from multiple levels.