Transformer-Ensemble-Based Implicit Spectral–Spatial Functions for Arbitrary-Resolution Hyperspectral Pansharpening

Arbitrary-resolution hyperspectral (ARHS) pansharpening seeks to enhance hyperspectral (HS) images to any desired spatial resolutions by integrating HS images with their connected panchromatic (PAN) images, which is obviously different from standard HS pansharpening where HS data are merely pansharpened to the spatial resolutions of PANs. Therefore, ARHS pansharpening has a great potential to broaden the applications of traditional pansharpening technique. However, ARHS pansharpening is faced with some inherent obstacles, including how to produce HS images with arbitrary resolutions beyond the training scales and how to maintain high spatial-spectral fidelities at any pansharpening scales. Implicit neural representations (INRs) is able to parametrically tune neural networks to accommodate unanalytical continuous characteristics of real-world scenes, and thus offer natural representations for ARHS pansharpening. The core of pansharpening INR design is to construct suitable implicit spectral-spatial functions (ISSFs). In this work, we propose two innovatively novel transformer-ensemble based ISSFs, i.e. pre-integrated ISSF (preISSF) and post-integrated ISSF (postISSF), which use suitable coordinate operation, novel ensemble feature extraction and coordinate query mapping to build strong spectral-spatial reconstruction capabilities, where some new tactics such as mode-1 transformer decomposition and explicit-implicit positional encoding system are involved. We test our preISSF and postISSF on three datasets. The experimental results verify the excellent performance of our methods.