S3ViT: self-supervised spectral vision transformer framework for hyperspectral unmixing

Hyperspectral unmixing aims to decompose each pixel in a hyperspectral image into a set of constituent endmembers and their corresponding abundances. Recent deep learning based approaches have demonstrated strong performance in capturing both spectral and spatial features. However, obtaining reliable per-pixel abundance ground truth in real hyperspectral scenes is generally infeasible, which motivates unsupervised and self-supervised unmixing strategies. In this work, we propose <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="m1"> <mml:mrow> <mml:msup> <mml:mrow> <mml:mtext>S</mml:mtext> </mml:mrow> <mml:mrow> <mml:mn>3</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> ViT, a self-supervised Spectral Vision Transformer designed for pixel-wise hyperspectral unmixing. The transformer captures spectral and spatial dependencies by applying self-attention over the full sequence of pixel tokens <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="m2"> <mml:mrow> <mml:mo stretchy="false">(</mml:mo> <mml:mrow> <mml:mn>1</mml:mn> <mml:mo>×</mml:mo> <mml:mn>1</mml:mn> </mml:mrow> <mml:mo stretchy="false">)</mml:mo> </mml:mrow> </mml:math> augmented with learnable positional embeddings. It operates without ground-truth annotations by generating pseudo labels through an unsupervised process: first, Singular Value Decomposition (SVD) is used to estimate the number of endmembers based on a thresholded singular value spectrum; then, k-means clustering provides cluster-derived priors that are used to form a contextual token and initialize spectral prototypes, without being treated as true abundance supervision. To guide training, two initialization tokens, one from Vertex Component Analysis (VCA) and one from the k-means cluster-derived priors, are embedded alongside patch tokens into the transformer. The model learns to estimate abundance maps while enforcing the Abundance Non-negativity and Sum-to-One constraints through ReLU and Softmax layers. Endmember spectra are later estimated from pixels with high predicted abundances. We evaluated <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="m3"> <mml:mrow> <mml:msup> <mml:mrow> <mml:mtext>S</mml:mtext> </mml:mrow> <mml:mrow> <mml:mn>3</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> ViT on the Samson, Jasper Ridge and Washington DC Mall benchmark datasets and compared it to state-of-the-art geometrical and deep learning methods. Our model achieves superior or comparable performance in both RMSE and SAD metrics, with up to 31% improvement in SAD and 25% in RMSE. These results indicate that a compact pixel-token ViT, guided by weak spectral priors and optimized via reconstruction losses, can achieve competitive unmixing performance on standard benchmarks.