From POC to production : how to build a support ticket triage system that survives contact with the real world

Support-ticket routing is a high-impact industrial text classification problem, yet models that perform well offline often fail to meet production requirements such as robustness to drift and reliable confidence estimates for selective automation. We study these challenges on a real B2B banking dataset of roughly 12k historical tickets labelled into 78 highly imbalanced routing classes. We benchmark transformer-based encoder classifiers against lightweight alternatives and find that sparse TF–IDF features with shallow learners can match or exceed tuned transformer accuracy while being faster to retrain and better at exploiting domain-specific lexical cues. We further improve performance via an ensemble re-ranking strategy that leverages strong top-k coverage and reaches 79.6% top-k accuracy for k = 1. Beyond accuracy, we evaluate calibration and stability under temporal drift: the final ensemble achieves a Brier score of 0.128 and ECE of 0.036, and retains 78.0% accuracy on a later-collected drift set spanning six months. Finally, we analyse LLM-assisted boilerplate removal and show that naive preprocessing can remove task-relevant signal, but a multi-view combination of raw and normalised text can yield gains. Overall, the results highlight practical modelling choices that enable robust, confidence-aware ticket triage in real deployments.