A regularized continuous-time hidden Markov model for identifying latent state transition patterns of poly-tobacco use

Hidden Markov models (HMMs) are widely used to characterize latent state transition patterns in substance use. However, traditional HMM frameworks are incompetent when dealing with the complexities introduced by high-dimensional risk factors and varying time intervals, particularly in determining the number of hidden states and selecting variables for state transition parameters. To tackle the analytical challenges in the Population Assessment of Tobacco and Health (PATH) Study, a nationally representative longitudinal cohort study on tobacco use, we propose a continuous-time HMM framework with a regularization algorithm to identify multi-dimensional risk factors underlying complex poly-tobacco use transitions. We develop an elastic-net regularization on the transition covariates to identify informative covariates and improve model estimation accuracy. The inclusion of key covariates enables accurate determination of the number of hidden states. We incorporate survey weights and information on strata and clustering throughout the modeling framework. We demonstrate the validity of our approach in determining state numbers, identifying informative covariates, and estimating model parameters through a series of simulations. Application of the proposed approach to PATH data analysis revealed several demographic, behavioral, and psychosocial factors that contribute to the differential risks of transition between tobacco-use states among youth and young adults. The model's capacity in identifying high-dimensional risk factors for underlying hidden variables substantiates its potential for enhancing public health research and informing interventions.