Operationalizing biological interpretability in health-related visual stimuli using multimodal artificial intelligence as standardized non-human raters

Abstract Evaluation and assessment research in health communication requires structured, construct-based approaches for interpreting visual materials. This study introduces the Asclepius Biological Interpretability Scale, developed through a two-stage modified Delphi process (a joint image-and-item panel followed by an independent item-only panel) grounded in the Biopsychosocial Model and Visual Literacy Theory, and examines whether multimodal artificial intelligence (AI) systems can reproducibly apply this framework relative to internally generated human reference scores. Six artworks (four health-themed and two controls) were evaluated by three multimodal AI models and four researchers (internal human reference) using a standardized protocol with independent researcher-operated accounts, memory-isolation procedures, repeated assessments over two days, and the final 20-item scale. Because Image Type varied across only six stimuli, health-themed versus control differences were summarized at the image level (Cohen’s d), human–AI differences were summarized descriptively at the rater level, and reproducibility was assessed using intraclass correlation coefficients (ICC) and Spearman correlations. Within this development image set, AI-generated scores were descriptively higher for health-themed than for control images on the three biomedical subscales (image-level Cohen’s d = 0.91–1.38) but not on Non-Medical Interpretability (d = 0.49); with four health-themed and two control images, these differences were not statistically significant ( p = 0.19–0.60). Reproducibility across researcher-operated accounts was excellent for the three biomedical subscales (ICC = 0.90–0.97) and moderate for Non-Medical Interpretability (ICC = 0.57), whereas agreement across assessment days was poor (ICC = 0.04–0.46). The four researchers’ internal reference scores were higher than the AI-generated scores on the biomedical subscales for health-themed images (mean differences 1.41–1.96 points) and lower on Non-Medical Interpretability (− 2.02 to − 2.19 points). These findings provide preliminary methodological evidence that multimodal AI can reproducibly apply a structured interpretability framework to controlled visual stimuli. Although day-to-day stability and the Non-Medical Interpretability subscale, which showed substantial internal-consistency problems, require further refinement, the results support the feasibility of using AI as standardized non-human raters within controlled methodological settings. Broader validation using authentic health communication materials and independent external human raters remains necessary.