Robust semi-automatic vessel tracing in the human retinal image by an instance segmentation neural network

Vasculature morphology and hierarchy are essential for blood perfusion. Human retinal circulation is an intricate vascular system emerging and remerging at the optic nerve head (ONH). Tracing retinal vascular branching from ONH can allow detailed morphological quantification, and yet remains a challenging task. We presented a robust semi-automatic vessel tracing algorithm on human fundus images by an instance segmentation neural network (InSegNN). InSegNN separates and labels individual vascular trees and enables tracing each tree throughout its branching. We have three strategies to improve robustness and accuracy: pseudotemporal learning, spatial multisampling, and dynamic probability map. We achieved 83% specificity, 50% improvement in symmetric best dice (SBD) compared to literature, and outperformed baseline U-net, and achieved 91% precision with 71% sensitivity. We have demonstrated tracing individual vessel trees from fundus images, and simultaneously retain vessel hierarchy information. InSegNN paves a way for subsequent analysis of vascular morphology in relation to retinal diseases.