Refusal, Confidence, and Explanation: A Cross-Modal Audit of Uncertainty-Expression Failures in Small Open-Weight Language and Vision-Language Models

Safety evaluations usually test refusal, abstention, and stated confidence separately. We treat them as one property, appropriate uncertainty expression, and audit it in text and vision across six small open-weight LLMs and five vision-language model (VLM) configurations. On a 240-item Korean benchmark, judged refusal is higher for harmful than for benign-looking requests in every model and under every labeling rule, but model-level orderings depend on the rule (primary rule: zephyr-7b-beta refuses harmful requests least, 42.5%, and benign ones most, 18.8%). Against 150 blinded labels from one annotator, the judges agree on refusal (κ=0.63) but overstate benign-prompt refusal (10.4% vs. 3.6%). Replacing sensitive words in benign prompts did not lower refusal (−1.9 pp [−5.4, 1.7]). Two VLMs abstain far more on out-of-distribution images, yet their stated confidence when they answer is equivalent within ±5 points to that on in-distribution images. On six MedMNIST v2 tasks, VLM accuracy is near chance (13.2–15.6% vs. 14.6% expected; ConvMixer reference on the same images: 67.8%) at ≈80% stated confidence, and stated evidence locations match CNN saliency only at chance level. Scaling from 2B to 8B shrinks the confidence gap without improving accuracy. At a fixed generation budget, only a step-by-step instruction to assess intent and risk raised benign-prompt refusal (+9.0 pp); generic step-by-step instructions did not. Neither scale nor chain-of-thought prompting resolves these failures.