Managing the effort-value tradeoff for integrating human judgment into surgery duration predictions

Abstract For surgery duration prediction, it has been shown that integrating human judgment into algorithmic prediction models results in a higher accuracy than machine learning based approaches can achieve alone. However, since humans are a scarce resource in healthcare, their involvement in surgery duration prediction has high opportunity costs as their time could be spent more effectively on patient care and other medical tasks. This raises the question of how these costs should influence human involvement in tasks where algorithms perform reasonably well at minimal cost. This paper introduces a framework for selectively integrating human judgment based on the costs of human involvement and prediction uncertainty, which is captured using algorithmic meta features. Our framework identifies instances in surgery duration prediction where human judgment should be integrated in algorithmic predictions and those where it should not. We evaluate our framework using a data set containing 70,610 surgeries from a large university hospital. We observe that while always integrating human judgment leads to an overall higher accuracy than never integrating human judgment, employing the framework can lead to an even higher accuracy with lower human workload. In our case study planning costs could be reduced by 27% compared to the current status quo. Our results indicate that a cost-sensitive allocation policy that excludes human judgment for task instances that are easy for the algorithm allows humans to focus on medical tasks.