Learning from Uncertainty: Improving Churn Prediction using Conformal Confidence Intervals

Churn prediction differs from other classification tasks due to its focus on successfully identifying positive cases within a top-ranked set of scored instances, rather than opti-mizing the overall performance on the entire population using classical accuracy-based metrics, as it is typically not viable for organizations to reach out to large customer groups and hence can only focus on a limited set of customers. Transitional techniques for enhancing the performance of top-ranking predictions usually emphasize on addressing the associated class imbalance problem by resampling data, applying ensemble models, or investigating cost-sensitive approaches to spur predictive models towards paying a higher emphasis on the detection of positive cases. This work shifts the attention from class distribution to-wards learning from uncertainty, by proposing a model-agnostic framework in which conformal prediction is applied to enhance the robustness of churn models by harnessing the confidence intervals of returned predictions. A comparative experiment, incorporating various real-life retention scenarios, demonstrates the effectiveness and superior performance achieved by our methodology.