Hijacking the “Safety Bifurcation Point”: Coupled Path Hijacking Attacks on Multimodal Vision–Language Models

The generalization of safety mechanisms from text-only LLMs to Large Vision–Language Models (LVLMs) remains poorly understood, especially concerning their vulnerability to typographic adversarial prompts. In this paper we ask where, inside a model, the safe-versus-unsafe decision is causally made. Focusing on decoder-only LVLMs, we show that safety is encoded as a non-linear, distributed representation that bifurcates early at a distinct Safety Bifurcation Point (SBP)—e.g., Layer 9 in LLaVA-1.5 and Layer 5 in Qwen-VL—and we expose a Correlation–Causality Gap: late layers are highly linearly separable yet causally inert. To leverage this insight we introduce a multistage, causal (interpretability-guided) framework to locate and verify safety representations, and we propose CPH (Coupled Path Hijacking), a white-box attack that jointly patches MLP and self-attention activations at the SBP to deterministically hijack the safe computational path. On the two models for which we report full tabulated results, CPH attains high oracle-scored attack success rates of 99.43% (LLaVA-1.5) and 98.57% (Qwen-VL); we additionally observe 99.71% on InstructBLIP-Vicuna, whose Vicuna backend is likewise decoder-only. We contrast this decoder-only vulnerability with encoder–decoder designs (e.g., a T5 backend or a Q-Former front-end), in which cross-modal grounding precedes decoding; we advance the greater robustness of such architectures as an architectural hypothesis for future validation, not as a tabulated result of this study. These findings indicate that VLM safety is architecture-dependent and motivate architecture-aware defenses for trustworthy multimodal AI.