On the Adversarial Vulnerabilities of Transfer Learning in Remote Sensing

Transfer learning with pretrained models from general computer vision tasks is widely adopted in remote sensing, offering reduced training costs and improved performance across applications like scene classification and semantic segmentation. However, this practice exposes downstream tasks to significant vulnerabilities, as adversaries can exploit publicly available pre-trained models to craft attacks that compromise model integrity. This paper introduces Adversarial Neuron Manipulation (ANM), a novel attack strategy with two variants—ANM-S (Single Neuron) and ANM-M (Multiple Neurons)—that generates highly transferable perturbations by selectively targeting fragile neurons in pretrained models. Unlike existing adversarial attacks, ANM requires no domain-specific knowledge, enhancing its applicability and efficiency across diverse settings. Evaluated on various benchmark datasets, ANM significantly degrades model performance. ANM-M, in particular, reduces accuracy by up to 84.24% in scene classification, 42.47% in semantic segmentation, and 75.33% in object detection (mAP). These results reveal critical weaknesses in deep learning models, especially their susceptibility to transferable attacks across CNN and Transformer architectures. By exposing the security risks inherent in transfer learning for remote sensing, ANM underscores the urgent need for robust defense mechanisms to safeguard safety-critical applications. Our findings advocate for enhanced model resilience and motivate future research into countering sophisticated adversarial threats.