Differentially private multimodal in-context learning via adaptive semantic manifold synthesis in large foundation models

Abstract Multimodal foundation models excel at in-context learning (ICL) but heavily rely on high-quality demonstrations. Using raw sensitive data for ICL in domains like healthcare and smart grids poses severe privacy risks. Furthermore, traditional differential privacy (DP) fine-tuning often degrades model utility and incurs prohibitive computational costs, especially in few-shot settings. To address this dilemma, we propose a novel, training-free Differentially Private In-Context Learning (DP-ICL) framework. At its core is an adaptive semantic manifold synthesis mechanism that shifts privacy protection from the parameter space to the semantic prompt space. By interpolating features and injecting calibrated noise along the hidden data manifold under a formal differential privacy guarantee, our framework substantially obfuscates sensitive micro-level attributes while preserving essential macro-level semantic class boundaries, an effect further corroborated by empirical resistance to membership inference and attribute inference attacks. Consequently, the foundation model's extensive prior knowledge effectively compensates for the DP noise-induced information loss. Extensive experiments on multimodal few-shot tasks demonstrate that DP-ICL successfully breaks the longstanding barrier between privacy protection and model utility. It achieves superior classification accuracy and robustness across a continuous spectrum of privacy budgets compared to state-of-the-art methods. Ultimately, this framework provides a secure, lightweight, and highly effective solution for deploying powerful AI models in data-sensitive industrial scenarios.