Abstract 2480: DeMixT 2.0: A deconvolution framework for sparse sequencing data using embedded negative binomial distribution

Abstract Background: Deconvolution methods have traditionally focused on estimating cell-type proportions in bulk tissues with mRNA profiles. They often overlook the cell transcriptional proportion (i.e., transcriptional activity), which reflects the cell functional states and could be important for understanding disease progression (e.g., tumor). Current methods face additional challenges when analyzing other RNA species, such as miRNAs, due to their low abundance and inherent sparsity. Similar limitations also affect spatial transcriptomics (ST) technologies, which enable in situ gene expression profiling but suffer from high data sparsity and limited resolution, with each spot mimicking a mini-bulk sample. Besides, methods for estimating transcriptional activity in ST data remain underdeveloped. Methods: To fill this gap, we extended our semi-reference-based DeMixT framework by incorporating a negative binomial (NB) distribution to account for sparse profiling data, namely DeMixNB. DeMixNB models the observed mixed expression Yig for gene g in sample i as the sum of tumor (Tig) and non-tumor (Nig) components, with each component following an NB distribution parameterized by gene-specific means, dispersion parameters, and sample-specific tumor transcriptomic proportions. The NB framework naturally accommodates the sparsity and overdispersion. DeMixNB runs in two tiers: it first leverages non-tumor reference to estimate the non-tumor component parameters, then uses an Iterated Conditional Modes (ICM) to jointly estimate tumor-specific parameters and tumor proportions in mixed samples. Results: We comprehensively validated DeMixNB from three aspects. First, we simulated data under various settings, including sample sizes, degrees of similarity between tumor and non-tumor components, and the number of distributions mixed in tumors. Across all scenarios, DeMixNB consistently achieved higher accuracy compared to existing methods. Then, to create a controlled experimental benchmark for miRNA analysis, we generated artificial mixtures using HS-5 fibroblasts combined with either wild-type or Dicer1 knockout HCT116 colorectal cancer cells at varying proportions. This design represents real tumor samples with known ground truth. DeMixNB maintained robust performance on sparse miRNA expression. To assess its broader utility, we applied DeMixNB to ST data from multiple cancer types. The estimated transcript proportions showed strong concordance with pathological annotations and revealed cell differentiation patterns not captured by conventional cell-type deconvolution methods. Conclusions: The DeMixT 2.0 framework, featuring DeMixNB, provides a robust solution for deconvolving sparse transcriptomic data. By accurately estimating tumor-specific transcriptional proportions, it enables deeper insight tumor heterogeneity at molecular and spatial levels. Citation Format: Hao Yan, Matthew Montierth, Liyang Xie, Peng Yang, Shuai Guo, Ruonan Li, Xiaoxi Pan, Caner Ercan, Yinyin Yuan, Kinga Németh, George A. Calin, Wenyi Wang. DeMixT 2.0: A deconvolution framework for sparse sequencing data using embedded negative binomial distribution [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 2480.