A frequency-decoupled framework for robust cross-center nuclei instance segmentation in histopathology

Reliable nuclei segmentation is essential for quantitative histopathology and precision oncology. However, models trained on single-center datasets often exhibit substantial performance degradation when applied to external institutions because of variations in H&E staining, scanner characteristics, magnification, and image resolution. Existing solutions commonly rely on stain normalization, access to target-domain data, or additional expert annotations, limiting their scalability in privacy-constrained and annotation-scarce multicenter settings. We developed FDS-HoVerNet, a robustness-oriented framework designed for annotation-efficient and cross-center nuclei segmentation without requiring stain normalization, site-specific adaptation, or target-domain supervision. The proposed framework incorporates a structure texture disentanglement mechanism to reduce domain-specific staining interference while preserving biologically relevant nuclear morphology. In addition, a multi-receptive-field aggregation module was introduced to enhance robustness against magnification and resolution heterogeneity. The model was trained exclusively on the single-center CoNSeP dataset and evaluated across six independent external datasets comprising diverse tumor types, staining characteristics, and acquisition settings. FDS-HoVerNet consistently demonstrated superior cross-domain generalization performance compared with existing baseline methods. On challenging multicenter datasets such as Kumar and MoNuSeg, the proposed framework improved the Aggregated Jaccard Index (AJI) from 0.279 to 0.550, corresponding to a 97% relative improvement over the baseline model, while approaching the performance of models trained using target-domain annotations. Qualitative analyses further confirmed that FDS-HoVerNet preserved critical nuclear structural characteristics under substantial staining variability, supporting reliable quantitative pathology analysis across institutions. FDS-HoVerNet provides a practical and scalable solution for robust cross-center computational pathology. By enabling reproducible nuclei segmentation without additional target-domain annotation or stain-specific calibration, the proposed framework may facilitate the deployment of interoperable AI systems in multicenter cancer research and precision oncology applications.