Uncertainty-aware variational soft prompt for few-shot tuning in fMRI-based mental state decoding

Functional magnetic resonance imaging (fMRI) is a widely leveraged signal in brain decoding, experiencing rapid advancement and demonstrating tremendous promises. The development of fMRI-based large models has piqued the interest of academics, who hope to overcome the existing limitations by taking advantage of advances in large model technology and increased accessibility of mental state datasets. This also raises the challenge of effectively transferring the pre-trained large models to new applications. To tackle this problem, this paper proposed a novel uncertainty-aware variational soft prompt approach to aid the pre-trained model in adapting to a new domain, while providing an estimation of the sample aleatoric uncertainty by using the reparameterization technique and associating the prompt vectors to uncertainty through the designed loss function. The comparative experiments showed that, the proposed approach can significantly improve model transfer performance and outperform the standard prompt method, in particular for dealing with few training samples. Moreover, the proposed approach demonstrated robustness to variations in model structure and the hyperparameter of the loss function, which also highlighted the practical usability of the proposed approach. The source code is publicly available at github.com/hzx-ntu/Uncertainty-aware-Variational-Soft-Prompt-for-Few-shot-Learning .