Discrete-time neural Markov models

Markov models are models used to describe movement of individuals between states. In medicine, Markov models are used to describe, for example, disease progression and the effects of interventions on such progression. The models can further be used to identify medical risk groups and aid clinicians in clinical decision making. Although accurate individual predictions are crucial for utility on an individual level, modeling of relationships between covariates and transition probabilities includes assumptions regarding the distribution of the data and the functional form of the relationship limiting accurate prediction of individual risks. The purpose of this work was to develop a flexible framework for discrete-time Markov models enabling the use of neural networks to predict individual transition probabilities. The framework was implemented in Julia. Both real-world data and simulated data including two possible competing transitions, censoring and non-linear covariate relationships was used for model evaluation. Three models were evalauted: a model excluding covariates, a model including linear mappings between covariates and transition probabilities, and a model using neural networks for predicting individual transition probabilities based on covariates. Models were trained by minimizing the negative likelihood using mini-batching of data for efficient computations of gradients. The neural network model outperformed the linear model in predicting individual transition probabilities. Both the negative likelihood and the mean squared error between observed and predicted individual transition probabilities were lower for the neural network model than for the linear model. Individual predicted transition probabilities were more evenly distributed around the line of identity for the neural network model than for the linear model indicating a better predicted functional form of the covariate mapping. A simple and flexible framework for development of discrete-time Markov models is described. Including neural networks to map the relationship between covariates and transition probabilities enables increased accuracy in predictions of individual risks and provides a tool to inform clinical interventions.