Abstract LB348: ProMiSЕ: Probabilistic multi-stain estimator for color separation in multiplexed brightfield histopathology images

Abstract Introduction: Brightfield histopathology uses chromogenic staining to assign distinct colors to different specimen elements. Multiplexing biomarker staining on the same slide has valuable advantages, including the ability to capture the spatial distribution of different cell populations and tissue structures. Color separation is a computational method that separates contributions from each chromogen. It enables the analysis of individual stain expressions, and serves as a crucial preprocessing step in numerous histopathology image analysis tasks (e.g. nuclear segmentation, stain normalization, cell classification). Most color separation methods produce accurate results for two stains, but tend to be significantly less reliable when applied to three or more stains. We introduce a new probabilistic method for stain separation, called Probabilistic Multiple Stain Estimator (ProMiSE), which estimates density maps of each stain at every point in the image and reliably handles larger numbers of stains. Method: Most prior work makes two limiting assumptions: first, that all pixels corresponding to a specific stain share a constant color-vector direction in the optical density space; and second, that the color of a pixel with mixed stains is a linear combination of the colors of the individual stains. In contrast, we model the variation in the color vectors of pixels corresponding to a pure stain using the Kent-Gaussian distribution, and the color distribution of mixed stain pixels as a convolution of the probability distributions of individual stains. Our use of the Kent-Gaussian distribution to model single stains is effective because both distributions accurately capture the statistics of color vectors that are concentrated around a mean direction in the optical density space, as commonly observed in brightfield images. Given an image and the number of stains (no other user input is needed), ProMiSE estimates parameters of the Kent-Gaussian distribution for each stain and parameters of the convolution of Kent-Gaussian distributions for mixed stains. These estimates are then used for localizing image regions with pure and mixed stains and after the decomposition of mixed stains into individual components, produce density maps. Results: Qualitative and quantitative evaluation has been done on images with six stains. Visual inspection indicates that previous methods tend to misclassify more saturated pure stains as mixtures, but that the proposed algorithm achieves good estimation in these regions. Quantitative evaluation was conducted by comparing the density maps generated by the proposed and five previous algorithms against ground truth derived from expert annotations. The similarity was evaluated using various types of standard metrics, such as mean-squared error. It shows significantly better performance of the proposed algorithm. Citation Format: Nikola Blagojević, Igor Mihajlović, Jovana Džunić, Aleksandar Randjelovic, Uroš Milivojević, Tijana Marjanovic, Dragan Janković, Misagh Naderi, Oanh Nguyen, Siniša Todorović, Ivana Mikić. ProMiSЕ: Probabilistic multi-stain estimator for color separation in multiplexed brightfield histopathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_2):Abstract nr LB348.