Test-time neural style transfer for domain harmonization, with application to cone detection in adaptive optics retinal imaging

Abstract Domain shift is a well-recognized challenge in medical image analysis using Deep Learning (DL), where variability in acquisition devices, operators, and patient populations can limit model generalization. Adaptive optics scanning laser ophthalmoscopy (AOSLO) imaging exemplifies this issue, as variations in image appearance across acquisitions can affect downstream quantitative analyses. In this work, we repurpose Neural Style Transfer as a test-time augmentation strategy (TT-NST) to harmonize image appearance between training and target domains without model retraining. Our NST approach is model-agnostic and can be applied to standard DL architectures. We evaluated performance, with and without TT-NST, using a well-established DL cone detector (MultiDimensional Recurrent Neural Network) across three datasets spanning far-domain, near-domain, and in-domain imaging conditions. TT-NST substantially improved mean Dice on the far-domain dataset from 0.43 to 0.72, primarily by increasing recall (0.34 to 0.75) with a limited reduction in precision (0.91 to 0.71). More modest gains were observed on the near-domain dataset, while performance slightly decreased on the in-domain dataset, as expected when test images match the training distribution. Compared with alternative test-time augmentation methods, TT-NST achieved the highest mean Dice on far-domain data. These results indicate that TT-NST is a label-free strategy for domain harmonization with potential to improve robustness across medical image analysis tasks under domain shift.