An Open-source Protocol for Deep Learning-based Segmentation of Tubular Structures in 3D Fluorescence Microscopy Images

Segmenting tubular structures in dense biological tissues from 3D fluorescence microscopy images is critical to study complex tissue but remains challenging due to image complexity, variability, and quality issues. Here, we introduce an open-source, user-friendly toolbox for end-to-end segmentation of tubular structures in 3D images, accessible to researchers without formal programming training. The toolbox features interactive Jupyter notebooks implementing two simple yet efficient deep learning architectures -- 3D U-Net and 3D U-Net with attention mechanisms -- for precise 3D segmentation of tubular networks. A key innovation is our simulation-based data augmentation strategy, which enhances model performance even with minimal training data (as few as one 3D image). Employing user-provided masks, the protocol generates artificial microscopy images with varying signal-to-noise ratios and simulates realistic imaging artifacts, including uneven staining, point spread function convolution, axial intensity variations, and Poisson and Gaussian noise. The protocol systematically guides users through data augmentation, model training, qualitative and quantitative evaluation on test sets, and inference on new images. We validate the toolbox by analyzing two morphologically distinct tubular networks in mouse liver tissue -- the bile canaliculi and sinusoidal networks -- demonstrating that both architectures perform well, with the attention U-Net slightly outperforming the standard U-Net when trained with augmented data. Our comprehensive toolbox, executable on local Graphics Processing Units (GPUs), high-performance computing clusters, or cloud platforms, contributes to the democratization of advanced image analysis for a broad spectrum of researchers.