Operationally calibrated open-world zero-day intrusion detection using hybrid conformal abstention

Abstract Network intrusion detection systems are often evaluated under closed-world assumptions, where every attack family observed during testing is already represented during training. This setting can overstate deployment readiness because operational environments routinely encounter unseen attack variants, evolving tactics, and uncertain traffic patterns. This paper proposes an operationally calibrated open-world intrusion detection framework for flow-based zero-day attack detection. The proposed method, Hybrid Tail-Calibrated Conformal Abstention (HTCCA), integrates complementary uncertainty and novelty indicators, including predictive entropy, margin uncertainty, maximum softmax probability uncertainty, prototype distance, Isolation Forest score, PCA reconstruction error, and extreme-value tail modeling. These scores are used in a two-stage decision process. First, a benign-calibrated UNKNOWN gate identifies traffic that deviates from the known operating space under a fixed false-positive-rate budget. Second, class-conditional conformal prediction assigns retained samples to known attack families or routes ambiguous cases to REVIEW. Experimental evaluation using leave-one-family-out zero-day protocols shows that entropy is the strongest single-score baseline at a 1% target benign false-positive rate, achieving 19.6% mean UNKNOWN-only detection. HTCCA-Conformal improves the broader unknown-alert pathway by combining UNKNOWN and REVIEW decisions, reaching 30.6% mean unknown-alert detection. Family-wise analysis shows stronger detection for high-volume denial-of-service patterns but weaker detection for Botnet, WebAttack, and BruteForce traffic. These findings highlight the importance of evaluating zero-day intrusion detection under operational false-positive constraints and analyst-centered triage settings.