PrivMLLM: Efficient Three-Party Multimodal Large Language Model Secure Inference Supported Prompt Privacy

Multimodal Large Language Models (MLLMs) represent the next frontier in artificial intelligence (AI), capable of processing and integrating diverse data types (text, images, audio and video) for richer understanding and generation. However, their potential is limited by privacy constraints: sensitive data resides with different owners who must keep it local, while MLLM parameters themselves require protection. We address this challenge by pioneering complete end-to-end privacy protection that simultaneously secures prompts, multimedia data, and MLLM parameters.In this work, we present PrivMLLM, the first general-purpose secure three-party (3PC) inference framework for MLLMs that serves two data owners while protecting both model parameters and inputs. Our approach is based on three key insights: (1) domain knowledge-aware optimization of fixed-point arithmetic for global performance gains, (2) hybrid secret-sharing protocols that reduce communication and rounds for linear/non-linear operations, and (3) constant-round function secret sharing (FSS)-based protocols for private embedding and maximum.We formally prove PrivMLLM’s security under the rigorous Universal Composability (UC) framework and demonstrate the first end-to-end secure inference system for MLLMs. Experimental results show that our solution enables secure inference for Llama3.2-11B-vision in under 0:5-minute per token, achieving 16 and 8 speedups compared to our implementations built with the SOTA 2PC (CrypTen<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup>) and 3PC (ABY<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3+</sup>) privacypreserving machine learning (PPML) frameworks respectively. Moreover, we benchmark the submodules ViT and LLM against the SOTA schemes, SHAFT (NDSS 2025) and SIGMA (PETS 2024), achieving 1:3~3× performance gains.