RMGAT: a Relational Multi-Scale Graph Attention Network for spatial transcriptomics domain identification with systematic component ablation

Abstract Background Mapping spatially coherent tissue domains from spatial transcriptomics data is a prerequisite for characterizing cell-type, developmental patterning, and disease-associated disruptions of tissue architecture. While graph neural network (GNN) methods have substantially improved domain identification over expression-only clustering, the effect of edge feature design (i.e. how spatial connections between tissue spots are represented) has never been systematically evaluated. All existing methods use a single scalar edge weight, leaving open the question of whether multi-dimensional edge features provide any benefit. Methods We introduce RMGAT, a Relational Multi-Scale Graph Attention Network that applies multi-dimensional learned edge features to spatial transcriptomics, to the best of our knowledge for the first time. Each spatial edge is described by an 8-dimensional learned feature vector. We combine this with a three-component training objective (graph reconstruction, NT-Xent contrastive learning, and a self-expression decoder) and a V30 post-processing pipeline using consensus clustering, Hungarian label alignment, and spatial refinement. Results Evaluated on the 12-section DLPFC (human dorsolateral prefrontal cortex) benchmark across 60 independent runs (5 seeds × 12 DLPFC sections), RMGAT achieves a mean adjusted Rand index (ARI) of 0.3422 ± 0.0402 (95% confidence interval [0.332, 0.353]), comparable to SpaGCN (~ 0.36). A systematic four-component ablation study with formal statistical testing identifies GAT attention as the only statistically confirmed essential contributor (ΔARI = − 0.028 when replaced with a graph convolutional network, GCN; p < 0.0001). Contrastive learning shows a directional benefit (ΔARI = − 0.017; p = 0.024 vs. scalar-weight baseline). The 8-dimensional EdgeMLP and self-expression decoder are both neutral (ΔARI ≈ 0.004; both p > 0.3), indicating neither contributes at PCA-50 input quality. Conclusions The primary contribution of this work is a systematic ablation study providing evidence-based design guidance for spatial transcriptomics GNNs. At PCA-50 input quality, GAT attention is the statistically confirmed critical component; contrastive learning is beneficial ( p = 0.024); multi-dimensional edge features and additional decoder objectives do not improve performance under this input regime. For practitioners, these findings support using GAT attention ( p < 0.0001) and contrastive learning ( p = 0.024, non-significant after Bonferroni correction), and investing in node feature quality over edge feature complexity.