Zero-shot performance of a concept-grounded chest radiograph foundation model depends on the evaluation dataset: a preregistered external evaluation of CLEAR and CheXzero

Abstract CLEAR, a concept-grounded vision-language model presented as auditable, scores chest radiographs through 368,294 radiological concepts. It reportedly outperformed CheXzero in zero-shot classification, but only on developer-chosen test sets, without a subgroup analysis or a systematic check of the concepts behind its zero-shot predictions. In a preregistered comparison on the NIH ChestX-ray14 test split (25,592 images, 2,797 patients), macro-AUROC over 14 findings was 0.695 for CheXzero, 0.635 for CLEAR's concept route and 0.700 for direct scoring with CLEAR's encoder. Neither model showed evidence of a sex gap (female minus male: CheXzero + 0.005, 95% CI − 0.015 to + 0.022; concept route − 0.008, − 0.029 to + 0.011) or an age gap under any of four prompt templates. Prompt wording alone reversed the ranking of CLEAR's direct route against CheXzero (− 0.038 to + 0.031). The concept space was nearly collapsed (participation ratio 2.95, against 11.18 for the image features), and one held-out-selected concept outperformed the concept route for every finding. On 15,000 VinDr-CXR radiographs the concept route led CheXzero by 0.057 (0.050 to 0.064), in the reported direction, but by only 0.023 where sex was recorded and 0.009 where age was recorded, the images carrying most of the disease. On the Indiana University collection, one of CLEAR's own test sets, it trailed CheXzero by 0.015 (0.007 to 0.024), using report indexing terms as labels. CLEAR's zero-shot advantage depended on the dataset, and the concept step added no accuracy over the image encoder. Developer-reported zero-shot gains and concept explanations need independent testing before health systems adopt them.