Deep Generative Modeling for Cognitive Diagnosis via Exploratory DeepCDMs

Abstract Deep generative modeling is a powerful framework in modern machine learning, renowned for its ability to use latent representations to predict and generate complex high-dimensional data. Its advantages have also been recognized in psychometrics. In this article, we substantially extend the deep cognitive diagnostic models (DeepCDMs) in Gu ( Psychometrika , 89:118–150, 2024) to challenging exploratory scenarios with deeper structures and all $\mathbf {Q}$ -matrices unknown. The exploratory DeepCDMs can be viewed as an adaptation of deep generative models (DGMs) toward diagnostic purposes. Compared to classic DGMs, exploratory DeepCDMs enjoy critical advantages, including identifiability, interpretability, parsimony, and sparsity, which are all necessary for diagnostic modeling. We propose a novel layer-wise expectation–maximization (EM) algorithm for parameter estimation, incorporating layer-wise nonlinear spectral initialization and $L_1$ penalty terms to promote sparsity. From a parameter estimation standpoint, this algorithm reduces sensitivity to initial values and mitigates estimation bias that commonly affects classical approaches for deep latent variable models. Meanwhile, from an algorithmic perspective, our method presents an original layer-wise EM framework, inspired by modular training in DGMs but uniquely designed for the structural and interpretability demands of diagnostic modeling. Extensive simulation studies and real data applications illustrate the effectiveness and efficiency of the proposed method.