Which domain fits best? domain similarity measures for two-step heterogeneous transfer learning for early laryngeal cancer diagnosis

• Proposes domain similarity measurements for two-step heterogeneous transfer learning, aiming to identify the most relevant heterogeneous intermediate domain for early laryngeal cancer detection. This approach is intended to serve as a guideline for selecting suitable intermediate domain prior to transfer learning, thereby improving its effectiveness and efficiency. • Investigates a wild range of commonly used medical imaging modalities as intermediate domains. • Validates the proposed similarity measurements by evaluating their alignment with transfer learning performance. • Analyzes factors affecting transfer learning performance, including intermediate domain size and the number of classes. Heterogeneous transfer learning is an effective approach for medical imaging problems with limited data and scarce public homogeneous resources, yet selecting the optimal domain for feature extraction remains an open, often intuition-driven challenge. This study proposes and validates a set of quantitative domain similarity measurements to a priori identify the most suitable intermediate domain for early laryngeal cancer detection within a two-step heterogeneous transfer learning (THTL) framework, thereby avoiding computationally expensive trial-and-error training. We introduce eight domain similarity measurements to access the similarity between intermediate domains and the target domain. Multiple common medical imaging modalities, including angiography, chest radiographs, lung computed tomography (CT), brain magnetic resonance imaging (MRI), pathological section images, diabetic retinopathy fundus images, skin lesion images, and gastroenteroscopy, are served as candidate intermediate domains. The resulting similarity scores are ranked and compared with actual THTL performance rankings. Finally, we employ normalized discounted cumulative gain (NDCG) to determine the most predictive measurement. Our findings reveal that Earth Mover’s Distance (EMD) is the most effective domain similarity measurement for grayscale images, while cosine similarity based on global features extracted from a convolutional neural network (CNN) is optimal for RGB images. Using these measurements, angiography and skin lesion images are identified as the most beneficial intermediate domains. This work establishes a validated, data-driven methodology that enables future researchers to replace subjective intuition in domain selection, thereby saving substantial computational resources while improving model performance.