Identifying Key Diagnostic Codes for Initial Belief State Prediction for Modeling Colorectal Cancer Screening Based on Reinforcement Learning

Colorectal cancer (CRC) is a leading cause of cancer-related deaths in the United States. Reinforcement learning (RL) allows modelling the dynamic decision-making in CRC screening and early detection as a partially observable Markov decision process (POMDP). Accurate computation of the initial belief state is a critical step in the POMDP implementation. We explore clinical codes that can contribute to the prediction of the initial belief state of the POMDP. Clinical codes from 4714 adults (2357 CRC and 2357 healthy) obtained by University of Kentucky HealthCare were employed. Each subject's demographics along with CRC related diagnostic codes including K (diseases of the digestive system), R (symptoms, signs, and abnormal clinical and laboratory findings), and Z (factors influencing health status and contact with health services) codes, were applied to compute the initial belief state using eXtreme Gradient Boosting (XGBoost), and the key factors were determined using Shapley Additive exPlanations (SHAP). A reasonable initial belief state computation was obtained, and we found that codes related to health maintenance and preventative care, screening for cancer and post-treatment, treatment and follow-up care, and digestive system condition positively contribute to the prediction of CRC cases. As these extracted key factors are ranked by importance, they can provide guidelines for selection of codes for the initial belief state computation in POMDP.