Multimodal fusion with hyperspectral images and RGB information for all-optical informed virtual staining

Histological analysis plays an irreplaceable role in tumor identification and disease development prediction. However, traditional histochemical approaches are inefficient due to their fussy, labor-intensive, and time-consuming workflows, potentially missing optimal treatment timing. In this paper, a deeply learned all-optical staining framework is proposed to achieve real-time, accurate, and plug-and-play pathology scans by introducing optical priors into the digital staining pipeline without any need for reagent staining. An end-to-end intermediate supervised optical-driven staining network (OSNet) is designed for automatic transformation from plain spectral transmittances to stained RGB images. Multi-scale convolution residual block (MultiResBlock) is specifically engineered to extract rich information from input spectral cubes. A pilot optical staining system is constructed to validate this approach using pairs of morphologically identical stained and unstained mice hepatoma slides. Experimental results demonstrate that the reconstructed H&E-stained images are visually equivalent to histochemically stained images, while the average scan time is just 3 minutes, over 13 times faster than conventional workflows. All-optical staining bypasses cumbersome histochemical procedures and offers results with high fidelity, addressing medical concerns related to interpretability and stability, which provides a new computationally generated panel for plain-to-stain processing with enormous potential in time-sensitive diagnosis, accurate treatment, and advanced molecular analysis.