A Fixed Budget Value Analysis of Few-Shot Medical Vision-Language Model Adaptation

Selecting between visual-only and language-informed adaptation is a moving target imaging informatics decision. The same scarce target labels train both the candidate strategies and provide evidence for choosing between them. We introduce a fixed budget, policy value framework for evaluating whether this choice is worthwhile. Four medical vision-language models (BiomedCLIP, PubMedCLIP, MedCLIP, and MedSigLIP) were evaluated across chest radiography and seven MedMNIST tasks at 1–16 shots per class, with external multilabel validation on NIH ChestX-ray14. We compared visual and language-anchored probes and evaluated label-free, support-informed, cross-validated, and held-out meta-selection rules. Policy value was defined as the macro-AUROC gain over always using the visual probe at the same annotation cost. At one shot, language anchoring was superior in 31% of encoder–task combinations, yielding an optimistic oracle opportunity of at most 0.026 macro-AUROC. No deployable selector demonstrated a practically meaningful positive held-out value. Selector value was non-positive at one shot and at most + 0.002 at four shots, with aggregate hierarchical confidence intervals including zero, with the external test demonstrating the same pattern. Harmful language choices were more frequent and larger than beneficial choices, so selection accuracy did not translate into policy value. Controlled class description interventions changed the sign and magnitude of the language effect, demonstrating a genuine yet encoder- and task-dependent opportunity. Under the evaluated fixed budgets, direct visual adaptation was the most dependable use of target labels. Imaging informatics evaluations should report policy value, negative transfer severity, and annotation cost, rather than only selection accuracy.

Authors

Institutions

Publication Details

Journal
Journal of Imaging Informatics in Medicine
Published
2026-11-19
DOI
https://doi.org/10.1007/s10278-026-02517-7
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Fixed Budget Value Analysis of Few-Shot Medical Vision-Language Model Adaptation

Panos Liatsis, Ali Zifan
Journal of Imaging Informatics in Medicine
Domain Adaptation and Few-Shot Learning
article

A Fixed Budget Value Analysis of Few-Shot Medical Vision-Language Model Adaptation

Panos Liatsis, Ali Zifan
article en

Abstract

Selecting between visual-only and language-informed adaptation is a moving target imaging informatics decision. The same scarce target labels train both the candidate strategies and provide evidence for choosing between them. We introduce a fixed budget, policy value framework for evaluating whether this choice is worthwhile. Four medical vision-language models (BiomedCLIP, PubMedCLIP, MedCLIP, and MedSigLIP) were evaluated across chest radiography and seven MedMNIST tasks at 1–16 shots per class, with external multilabel validation on NIH ChestX-ray14. We compared visual and language-anchored probes and evaluated label-free, support-informed, cross-validated, and held-out meta-selection rules. Policy value was defined as the macro-AUROC gain over always using the visual probe at the same annotation cost. At one shot, language anchoring was superior in 31% of encoder–task combinations, yielding an optimistic oracle opportunity of at most 0.026 macro-AUROC. No deployable selector demonstrated a practically meaningful positive held-out value. Selector value was non-positive at one shot and at most + 0.002 at four shots, with aggregate hierarchical confidence intervals including zero, with the external test demonstrating the same pattern. Harmful language choices were more frequent and larger than beneficial choices, so selection accuracy did not translate into policy value. Controlled class description interventions changed the sign and magnitude of the language effect, demonstrating a genuine yet encoder- and task-dependent opportunity. Under the evaluated fixed budgets, direct visual adaptation was the most dependable use of target labels. Imaging informatics evaluations should report policy value, negative transfer severity, and annotation cost, rather than only selection accuracy.

Journal of Imaging Informatics in Medicine
Khalifa University of Science and Technology (AE), University of California San Diego (US)
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon