Unifying Low-Resolution and High-Resolution Alignment by Event Cameras for Space-Time Video Super-Resolution

Event cameras deliver asynchronous pixel intensity changes, which result in sparse event data that offers the advantages of high temporal resolution. These high temporal characteristics make researchers naturally incorporate event cameras into video frame interpolation (VFI) and video super-resolution (VSR). In this paper, we make the first attempt to solve the space-time video super-resolution (STVSR) task effectively, addressing both VFI and VSR simultaneously, by leveraging temporally dense events. STVSR aims to generate intermediate high-resolution (HR) videos between consecutive low-resolution (LR) frames. To fully exploit the high temporal frequency of events for STVSR, we focus on temporal alignment in two stages, at low-resolution and after up-sampling in high-resolution. In temporal alignment at low-resolution, to upsample spatial dimensions effectively, we leverage high temporal features to preserve spatial context. On the other hand, for temporal alignment at the high-resolution stage, we employ a deformable sampling process from events to achieve accurate alignment with forward and backward directions. In addition, we provide the SuperREST dataset, which features high-frequency details and complex motion in an RGB-Event setup. Experimental results on several datasets demonstrate that our method achieves a significant performance gain on STVSR tasks with low computational cost. Our codes and datasets are available at h t t ps: //github.com/Chohoonhee/ESTNet.