stRGAT: identifying spatial domains in spatial transcriptomics via a relational graph attention network

Spatial transcriptomics has rapidly advanced the study of tissue architecture by integrating gene expression profiles, spatial coordinates, and high-resolution histology images. However, accurately identifying spatial domains with coherent transcriptomic and morphological patterns remains challenging. We developed stRGAT, a deep learning framework that integrates spatial information and gene expression to learn low-dimensional latent embeddings for spatial domain identification. stRGAT adaptively models the similarity among neighboring and cross-regional spots, enabling coherent spatial partitioning. Additionally, the framework can be extended to detect spatially variable genes that reflect biologically meaningful spatial expression patterns. We conducted extensive evaluations on cross-species datasets, continuous tissue sections, and fine-grained mouse brainstem datasets. Quantitative comparisons against current mainstream methods demonstrate that stRGAT more accurately identifies spatial domains, achieving higher clustering agreement metrics and clearer domain boundaries. Furthermore, stRGAT’s ability to capture spatially variable genes underscores its versatility in spatial transcriptomic analysis. stRGAT provides a robust and generalizable approach for spatial domain delineation and associated tasks in spatial transcriptomics, offering improved performance over existing methods and broad applicability for studying tissue organization and function.