Comparative Analysis of Clinical Finding Retrieval Difficulty in Chest X-ray Retrieval Models

Abstract Sentence-level retrieval generates chest X-ray (CXR) reports by selecting clinical sentences from a predefined pool. Retrieval performance depends on the composition of this pool, yet aggregate metrics can conceal differences between clinical findings. This study examined whether finding-level retrieval difficulty reflects model limitations or retrieval pool composition. Three sentence-level models (a JoImTeRNet-based model developed by the authors, CXR-RePaiR and CXR-ReDonE) retrieved the top 10 sentences for 10,000 frontal CheXpert images. Sentences were drawn from an original pool of 129,906 MIMIC-CXR sentences, 30 balanced pools of approximately 6,200 sentences each (up to 450 positive sentences per finding, drawn with different random seeds) and 50 random pools of equal size. Sentences were labeled with the CheXbert labeler. Micro-averaged recall, precision and F1, finding-level recall at K=5 and cross-model Spearman correlation were evaluated. Balancing raised micro-recall for every model at every K and finding-level recall for 13 of 14 findings, whereas No Finding recall fell by 0.119 to 0.187 (mean of 30 balanced pools). For 36 of 42 finding-model combinations the change exceeded that of random pools in at least half of the balanced pools, and at K=5 balanced-pool recall exceeded the random-pool mean by 0.07 to 0.08. Precision decreased and F1 varied. Cross-model agreement across 13 findings declined from ρ=0.687–0.808 (original pool) to ρ=0.153–0.577 (balanced pools), while random pools gave ρ=0.712–0.788; with only 13 findings, this decrease should be regarded as descriptive. Lung Lesion remained among the five lowest-recall findings for all three models under the original pool and in all 30 balanced pools. Retrieval pool composition influences finding-level recall and apparent difficulty in CXR sentence retrieval, and part of the difficulty observed under the full pool is pool-dependent. Evaluations should report pool composition and finding-level results alongside aggregate scores.