An Empirical Study of Parameter-Efficient Fine-Tuning in Vision-Language Model Safety Pipelines
Vision–language models (VLMs) face safety risks from typographic visual prompt injection, where adversarial text embedded in images can influence model behavior. When several protection layers are combined, upstream filtering changes the residual attack set seen by the downstream model. This thesis studies a two-layer pipeline combining an OCR-based input guardrail with LoRA-based fine-tuning, focusing on the additional contribution of model-side adaptation after upstream filtering. A controlled 2 × 2 factorial study on Qwen2-VL-7B uses CyberSecEval3 for adversarial evaluation and TextVQA for benign utility. The evaluation is extended to three additional VLMs under a common protocol, with MM-SafetyBench as a second safety benchmark. The pipeline records guardrail decisions, risk scores and model responses at the sample level. On the primary model, overall attack success rate decreases from 48.75% (Baseline) to 18.00% (Guardrail-only), 39.75% (LoRA-only) and 14.25% (Guardrail + LoRA). Among the 164 samples that pass the guardrail, LoRA reduces attack success by approximately 9.15 percentage points; at the complete-system level the additional decrease is 3.75 percentage points. The benefits are task- and setting-dependent. TextVQA accuracy remains nearly unchanged, but on the held-out task evaluation, Size accuracy under attack is 98% for Base and 76% for LoRA, so the aggregate attacked-task advantage of LoRA cannot be interpreted entirely as a defense benefit. The guardrail blocks none of the 500 TextVQA or 168 MM-SafetyBench samples, but falsely blocks 55% in a trigger-enriched stress set. Cross-model results further show substantial variation in baseline vulnerability. Overall, input-side filtering and model-side adaptation provide partially complementary protection, but the observed benefits depend on the task, benchmark and input distribution.
Authors
- Jilong Li
Publication Details
- Journal
- Gothenburg University Publications Electronic Archive (Gothenburg University)
- Published
- 2026-11-12
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00