Benchmarking locally deployed open-weight vision–language models on the Japanese National Examination for Pharmacists

Abstract Objective We evaluated the performance of locally deployable open-weight vision–language models (VLMs) in the 2026 Japanese National Examination for Pharmacists and examined the influence of model scale and generational differences on domain-specific examination outcomes. Twenty-six open-weight VLMs were evaluated under standardized execution conditions in three independent trials. Results Accuracy varied across models, with qwen2.5-vl:72b achieving the highest accuracy at 62.8% (± 0.17%). Performance differed across subject domains: pathophysiology/drug therapy, biology, and pharmacology showed high accuracy, whereas chemistry showed lower performance across all architectures and parameter scales. This pattern may partly reflect challenges associated with specialized scientific images, including chemical structures. The model parameter count showed a moderate positive association with the overall accuracy (R² = 0.59), indicating a scaling trend in which larger models achieved higher accuracy scores. Within the model families, accuracy generally tended to increase with the model size. However, substantial performance variability was observed among the models, particularly among models with fewer than 14B parameters, whereas recently released models appeared to exhibit a more consistent scale–performance relationship. These findings suggest that factors beyond the model scale, including architectural and training differences, may contribute to performance variation. This study provides a benchmark for open-weight VLMs in the pharmaceutical domain.