Diagnostic probes reveal shortcuts induced by the benchmark protocol in vision-language model annotation pipelines for video-based self-harm detection

Vision-language models (VLMs) are increasingly used as annotators: a model captions clinical images or video, a classifier is trained on the captions, and the resulting benchmark score is read as evidence of clinical capability. We describe two diagnostic probes that test whether such a score depends on the target finding. Necessity deletes the most class-discriminative caption tokens; sufficiency classifies from content-agnostic surface-form statistics alone. On an acted self-harm video benchmark (5,183 clips; seven actors; leave-one-subject-out evaluation), four VLM pipelines with a clinically engineered prompt reached F1 0.784–0.944 under the benchmark protocol. Deleting the 40 most discriminative tokens retained 92–100% of F1, and eight surface-form statistics recovered 83–95%. Re-analysis showed that this surface-form signal arose mainly from the protocol, which trims boundary segments from normal clips only and, with our captioning order, changes the number and position of captioned frames, and the prompt template, by class. With label-independent frame selection, AUC fell from 0.89–​0.99 to 0.53–0.65, and three pipelines retained a modest signal above a permutation null. The probes thus flagged a score that rested largely on a protocol artifact. Benchmark scores of VLM-annotator pipelines should be checked in this way before they are given clinical weight.