Optimizing deep learning for brain vessel segmentation in CT angiography: a study on domain adaptation and generalizability

Accurate segmentation of intracranial vasculature from CT angiography (CTA) is critical for cerebrovascular disease research and clinical applications. Deep learning-based models, such as DeepMedic and nnU-Net, offer promising solutions, but their generalizability across datasets remains a challenge. In this study, we evaluated transfer learning strategies for segmenting brain vessels from CTA images acquired from different institutions. We investigated the performance of DeepMedic and nnU-Net in fine-tuning experiments, assessing the impact of layer freezing and learning rate adjustments. Our results demonstrated that freezing the initial two layers in DeepMedic’s low-resolution pathway provided optimal performance improvements when fine-tuning to an external dataset. Additionally, nnU-Net, even without layer freezing, exhibited superior generalizability, with fine-tuning yielding significant performance gains in segmenting smaller vessels. However, in a separate external dataset (TopCoW), fine-tuning led to performance degradation compared to training from scratch, highlighting the impact of task differences on transfer learning efficacy. Specifically, differences in annotation strategies and vessel segmentation coverage contributed to the observed decline, emphasizing the importance of dataset-specific adaptations in transfer learning. Overall, our study highlights the potential and limitations of transfer learning in cerebrovascular segmentation tasks. While fine-tuning improved performance in similar datasets, significant task discrepancies necessitated training from scratch. These findings provide insights into optimizing deep learning strategies for robust and generalizable cerebrovascular segmentation.