Distinct morphological features of malignant cells in large B-cell lymphoma identified by soft-segmentation in imaging-based spatial proteomics

Abstract Background: Mature B-cell neoplasms display densely packed and morphologically heterogeneous cells, often complicating tissue-level analysis. Recent studies have revealed the complexity of the tumor microenvironment (TME) in these malignancies (Roider, Nature Cell Biology 2024; Colombo, Blood Advances 2022), providing insights into prognosis and treatment response. Spatial profiling platforms such as imaging mass cytometry (IMC) enable single-cell, in situ analysis using multiplexed antibody panels, but accurate and interpretable segmentation remains a challenge in lymphoid tissues where cells frequently overlap, deform, and exhibit non-canonical marker profiles. Common tools, including DeepCell Mesmer and Cellpose, assume round, non-overlapping cells with clear boundaries, a simplification that often fails in DLBCL, FL, and cHL (Dufva et al. 2020). These limitations can yield biologically misleading segmentations and restrict precise TME characterization. Methods: To address this gap, we developed SegMO (Segmentation and Morphological feature Optimization), an unsupervised, annotation-free segmentation framework compatible with multiple spatial proteomics platforms. SegMO integrates nucleus-anchored seeding, probabilistic modeling of marker expression, cluster-guided refinement, and a reinforcement learning (RL) module to produce biologically realistic, soft whole-cell masks that allow partial overlaps and irregular shapes. Protein expression is modeled with flexible statistical distributions to accommodate partial or co-expressed markers, and cells are clustered by expression features before iterative mask refinement to improve marker purity and spatial coherence. The RL model optimizes boundaries through pixel-level adjustments to maximize cell fidelity, recover low-signal regions, and capture realistic morphology. Results: We applied SegMO to 24 IMC samples (3 tonsil, 2 spleen, 9 brain, 10 lymphoma) stained with 43 protein markers, from a larger cohort representing 76 individuals (DLBCL, FL, cHL, and healthy donors). SegMO segmented ~74,000 cells and grouped them into biologically distinct clusters based on marker-driven statistical modeling. Compared to Mesmer, SegMO improved nuclear signal integrity (0.92 vs. 0.90, P < 1×10⁻⁵), detected fewer weak-signal nuclei (<0.1% vs. 2.5%, P < 10⁻⁴), and generated more accurate boundaries in high-density and overlapping regions. Soft segmentation preserved subtle morphological features such as protrusions, compression, elongation, and partial overlaps that rigid models did not capture. Morphological profiling showed that Cluster 7, enriched in tumor tissues and likely neoplastic B cells, had one of the largest average cell sizes (mean = 139.95 pixels²) and highest nuclear ratios (mean = 0.65), both significantly greater than Cluster 4 (fibroblasts; mean = 97.6, ratio = 0.59) and Cluster 5 (CD4⁺ T cells; mean = 101.6, ratio = 0.56) from healthy tissue (ANOVA P < 1×10⁻⁶³ for size, P < 1×10⁻⁹¹ for ratio; Tukey HSD P < 0.001). Cluster 1, composed of CD4⁺CD7⁺ lymphocytes, was enriched in the GCB subtype (Wilcoxon P = 0.006), with intermediate size (mean = 127.70), high nuclear ratios (mean = 0.61), and irregular shapes (elongation, asymmetry, or compression) reflecting activation, mobility, and spatial pressure in the TME. These patterns were not detected by standard segmentation tools. Beyond cell-level statistics, SegMO enabled spatial visualization of marker-defined clusters and uncovered architectural features including immune–tumor colocalization, irregular clustering, and localized crowding in aggressive lymphomas. The RL module further improved segmentation in low-signal or noisy regions, recovering masks otherwise lost or poorly defined. Conclusion: SegMO is a robust, versatile, annotation-free AI framework for single-cell segmentation in multiplex tissue imaging of hematologic samples. By integrating probabilistic modeling, cluster-guided refinement, and reinforcement learning, SegMO produces biologically realistic boundaries in complex TMEs, enhancing both morphological fidelity and marker signal accuracy. This capability supports high-resolution spatial and morphological analysis, enabling precise characterization of tumor-associated features, immune interactions, and proliferative states in lymphomas and other hematologic malignancies. Full-cohort analysis with integrated RL, revealing additional morphological patterns and TME phenotypes, will be presented.