An Empirical Study of Parameter-Efficient Fine-Tuning in Vision-Language Model Safety Pipelines

Vision–language models (VLMs) face safety risks from typographic visual prompt injection, where adversarial text embedded in images can influence model behavior. When several protection layers are combined, upstream filtering changes the residual attack set seen by the downstream model. This thesis studies a two-layer pipeline combining an OCR-based input guardrail with LoRA-based fine-tuning, focusing on the additional contribution of model-side adaptation after upstream filtering. A controlled 2 × 2 factorial study on Qwen2-VL-7B uses CyberSecEval3 for adversarial evaluation and TextVQA for benign utility. The evaluation is extended to three additional VLMs under a common protocol, with MM-SafetyBench as a second safety benchmark. The pipeline records guardrail decisions, risk scores and model responses at the sample level. On the primary model, overall attack success rate decreases from 48.75% (Baseline) to 18.00% (Guardrail-only), 39.75% (LoRA-only) and 14.25% (Guardrail + LoRA). Among the 164 samples that pass the guardrail, LoRA reduces attack success by approximately 9.15 percentage points; at the complete-system level the additional decrease is 3.75 percentage points. The benefits are task- and setting-dependent. TextVQA accuracy remains nearly unchanged, but on the held-out task evaluation, Size accuracy under attack is 98% for Base and 76% for LoRA, so the aggregate attacked-task advantage of LoRA cannot be interpreted entirely as a defense benefit. The guardrail blocks none of the 500 TextVQA or 168 MM-SafetyBench samples, but falsely blocks 55% in a trigger-enriched stress set. Cross-model results further show substantial variation in baseline vulnerability. Overall, input-side filtering and model-side adaptation provide partially complementary protection, but the observed benefits depend on the task, benchmark and input distribution.

Authors

Publication Details

Journal
Gothenburg University Publications Electronic Archive (Gothenburg University)
Published
2026-11-12
Primary Topic
Adversarial Robustness in Machine Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

An Empirical Study of Parameter-Efficient Fine-Tuning in Vision-Language Model Safety Pipelines

Jilong Li
Gothenburg University Publications Electronic Archive (Gothenburg University)
Adversarial Robustness in Machine Learning
article

An Empirical Study of Parameter-Efficient Fine-Tuning in Vision-Language Model Safety Pipelines

Jilong Li
article en

Abstract

Vision–language models (VLMs) face safety risks from typographic visual prompt injection, where adversarial text embedded in images can influence model behavior. When several protection layers are combined, upstream filtering changes the residual attack set seen by the downstream model. This thesis studies a two-layer pipeline combining an OCR-based input guardrail with LoRA-based fine-tuning, focusing on the additional contribution of model-side adaptation after upstream filtering. A controlled 2 × 2 factorial study on Qwen2-VL-7B uses CyberSecEval3 for adversarial evaluation and TextVQA for benign utility. The evaluation is extended to three additional VLMs under a common protocol, with MM-SafetyBench as a second safety benchmark. The pipeline records guardrail decisions, risk scores and model responses at the sample level. On the primary model, overall attack success rate decreases from 48.75% (Baseline) to 18.00% (Guardrail-only), 39.75% (LoRA-only) and 14.25% (Guardrail + LoRA). Among the 164 samples that pass the guardrail, LoRA reduces attack success by approximately 9.15 percentage points; at the complete-system level the additional decrease is 3.75 percentage points. The benefits are task- and setting-dependent. TextVQA accuracy remains nearly unchanged, but on the held-out task evaluation, Size accuracy under attack is 98% for Base and 76% for LoRA, so the aggregate attacked-task advantage of LoRA cannot be interpreted entirely as a defense benefit. The guardrail blocks none of the 500 TextVQA or 168 MM-SafetyBench samples, but falsely blocks 55% in a trigger-enriched stress set. Cross-model results further show substantial variation in baseline vulnerability. Overall, input-side filtering and model-side adaptation provide partially complementary protection, but the observed benefits depend on the task, benchmark and input distribution.

Gothenburg University Publications Electronic Archive (Gothenburg University)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon