Does the Pneumonia Model Look at the Lungs? Shortcuts in a Pediatric Chest X-ray Benchmark and a Lung-Crop Fix Tested at Other Sites

The pediatric chest X-ray dataset of Kermany et al. is one of the most widely used benchmarks for detecting pneumonia with deep learning, and models trained on it routinely report test AUCs above 0.9. I asked whether such a model actually uses the lungs. Training ResNet-18 on images in which the lungs were hidden, I found that it still separates pneumonia from normal scans with a test AUC of 0.876 ± 0.029 (5 seeds), and a logistic regression on the image's width and height alone reaches 0.899. The non-lung signal is specific to the source hospital: on 3,256 chest X-rays of children from the NIH ChestX-ray14 dataset, the same lungs-hidden models fall to chance once the AP/PA view is controlled for (within-view AUC 0.53), and image size falls to 0.57. Models that see the lungs also lose accuracy at the new site but stay well above chance. I then tested a simple fix that forces the model onto the lungs: crop each image to the lung region found by a pretrained segmentation model, apply contrast-limited histogram equalization, and add standard augmentation. Across three architectures (ResNet-18, DenseNet-121 and a DenseNet-121 pretrained on CheXpert), the fix raised within-view external AUC over the full-image model by +0.05 to +0.09, both on NIH labels mined from reports and on 677 of the same children relabeled by radiologists in the RSNA Pneumonia Challenge, and it did so on every seed. It did not help on a small set of phone photographs of X-ray films from Bangladesh, where lung segmentation often failed. A high internal test score on this benchmark says little about whether the model has learned pneumonia; testing with the lungs hidden and at another site is a cheap way to find out. The attached archive contains the code, per-run results (CSV) and figures. The datasets are public and are not redistributed.