XGB-UNetFuse: A density-aware hybrid framework for automated retinal ganglion cell quantification

Accurate quantification of retinal ganglion cells (RGCs) is critical for assessing neurodegeneration and evaluating therapeutic interventions in experimental glaucoma. Manual RGCs quantification is time-consuming, subjective, and inconsistent, underscoring the need for automated and scalable approaches that can adapt to varying cell densities and imaging conditions. We present XGB-UNetFuse (Fusion of Extreme Gradient Boosting (XGB) classifier with U-Net and Light-U-Net models) for automated RGCs quantification from microscopy images. The workflow begins with dataset preparation, including conversion of standard microscopic images into differential interference contrast (DIC)-like representations to enhance cell boundary visibility and contrast during preprocessing. Feature vectors are extracted and combined with manual ground truth to perform binary classification of whole images into low or moderate/high-density categories using an XGBoost model. Based on this classification, the system adaptively selects between two independently trained detection networks: Light-U-Net for low-density images and U-Net for moderate/high-density images. The outputs undergo tile reconstruction, Gaussian smoothing, intensity thresholding, and local-maxima detection to generate precise cell counts. Experimental validation across training, test, and an independent evaluation datasets demonstrated that XGB-UNetFuse achieved superior accuracy and generalization, yielding the lowest mean counting error and highest agreement with manual counts. Bland–Altman analysis confirmed a mean bias within ± 10% across all retinal regions. The proposed density-aware adaptive framework provides a robust, scalable, and reproducible solution for high-throughput RGCs quantification, advancing automated retinal microscopic analysis and translational ophthalmic research.