Deep Learning Based Clustering of Gene Expression Data in Cancer Patients using Variational Autoencoder Model

Background: High mortality associated with cancer presents challenges in both diagnosis and treatment. Deep learning-based clustering analysis provides a powerful tool for identifying molecular subtypes within cancer enabling personalized treatment strategies. Objectives: This project has two objectives: (1) to cluster gene expression data for cancer patients into unique subtypes to uncover novel biological insights and improve patient outcomes; (2) to identify unique molecular signatures driving each cluster. Methods: GDC Pan-Cancer data, consisting of 33 tumor types with 11,506 samples and 32,967 gene expression features, from TCGA repository was used. Outliers were identified using interquartile criterion, and replaced using multiple imputation by chained equations. A variational autoencoder (VAE) model was used to extract latent representations within the gene expression data. These latent features were then utilized in K-Means clustering to identify distinct clusters based on similarity in gene expression profiles. Survival rate and median survival time for each cluster were computed, and statistical tests were used to determine the significance of the observed differences. Finally, gene signatures for individual clusters were developed using standardized mean difference (SMD) analysis, determining features significant in one cluster vs the others (p<0.05). Results: Survival rates vary significantly across clusters (p<0.05). Clustering helped identify clusters with low survival rates. Subsequently, distinct gene signatures identified for each cluster helped determine molecular mechanisms driving survival outcomes. Conclusion: A deep learning-based model efficiently groups cancer patients, revealing unique molecular signatures within each subgroup. Screening and prioritizing treatment for patients in subgroups with lower survival rates could enhance patient management.