Spatialsmooth: a spatially-aware convolutional autoencoder framework for enhanced deconvolution of spatial transcriptomics data

Spatial transcriptomics technologies have revolutionized our understanding of tissue heterogeneity by providing gene expression data with spatial context. However, due to the limited resolution of some of these techniques, multiple cell types are often captured at each spatial location. Several spatial deconvolution methods have been developed in recent years to infer cell type compositions from these mixed spots, but their results can be noisy and spatially inconsistent. In this study, we introduce a spatial smoothing method, Spatialsmooth based on convolutional autoencoder, in which several spatial deconvolution tools are integrated to obtain the deconvolution results, the spatial location information is fully utilized using positional encoding, and the deconvolved cell type compositions are smoothed using a convolutional autoencoder to optimize the distribution of different cell type compositions in space. This method integrates multiple spatial deconvolutions tools and fully utilizes spatial location information and cell type composition inferred by multiple deconvolution algorithms to produce smooth, biologically plausible cell type distributions. The smoothing effect was measured by using three spatial metrics, namely, Moran's I, Geary's C, and Total Variation, on multiple pairs of datasets. pancreatic ductal adenocarcinomas (PDAC) dataset, Spatialsmooth reached 0.52 on the Moran's I metric (92% higher than the Redeconve method), the Geary's C metric was reduced to 0.47 (a 45% improvement compared to RCTD), and Total Variation was reduced by 22%. Applied to benchmark data, multiple cell types and molecular markers with different spatial localizations were identified, closer to the spatial projection of marker genes than other deconvolution tools. ( https://github.com/njjyxl/Spatialsmooth ).