Probabilistic representation learning of baseline transcriptomes enables prediction and pathway-level interpretation of anticancer drug responses

Abstract Baseline transcriptomes can predict anticancer drug response, but their high dimensionality complicates pathway-level interpretation. We developed a probabilistic representation learning framework using a reusable variational autoencoder (VAE) latent space to prioritize response-associated factors by aggregate importance and drug preferentiality. Across five reference drugs, leakage-free repeated cross-validation showed that gene-level Ridge regression performed best, whereas 64-dimensional latent representations retained predictive signal after compression. In a separate interpretation VAE fitted to 939 cancer cell lines, six aggregate-importance candidates were bootstrap-robust within the fixed representation and were associated with immune/inflammatory, oncogenic, cell-cycle, stress-response, and cellular-processing programs. Drug-preferential candidates lacked factor-level bootstrap robustness, although several pathway associations remained supported after lineage adjustment. Five independent VAE refits showed partial gene-program concordance but seed-sensitive axes, while a principal component analysis baseline supported stable factor selection and pathway annotation. As an exploratory application, baseline transcriptomes from 15 QAL333-tested cell lines were projected into the fixed VAE without encoder retraining. QAL333-prioritized factors were associated with inflammatory, oncogenic, cellular-processing, and microenvironment-related pathways. The two-model zebrafish xenograft comparison remained descriptive and hypothesis-generating. These findings support reusable, stability-aware latent-space analysis as a framework for pathway-level hypothesis generation from high-dimensional transcriptomic drug-response data.