Conformal prediction for network intrusion triage and analysis of unseen attack families

High benchmark accuracy does not inform a security operations center which alerts to trust or whether attacks absent from training can be flagged. We evaluated conformal prediction as an analyst-triage layer and compared probability and feature-space novelty scores across 15 leave-one-attack-family-out experiments on UNSW-NB15 and CIC-IDS2017. Primary analyses use the first-occurrence feature deduplication. Novelty performance was family dependent and sometimes inverted: four families had probability-space area under the receiver operating characteristic curve (AUROC) of at least 0.70, while the nearest-neighbor AUROC was at most 0.45, including web attacks (0.966 vs. 0.298), whereas port scanning reversed the ordering (0.166 vs. 0.812). The benign-only isolation forest was weaker overall; a matched forest fitted on all known classes also remained family dependent. Marginal conformal coverage was near 0.900, whereas one class fell to 0.295. Class-conditional calibration raised every seed-averaged class coverage to at least 0.894, including the class with 35 calibration examples. Its efficiency depended strongly on the base classifier: Rule B auto-dispatch was 0.3% for gradient boosting on UNSW-NB15 but 70.4% for a random forest. Calibration-selected probability thresholds often achieve similar or lower selective risk, whereas among the methods compared, only the conformal constructions supplied finite-sample coverage control. The conformal correction modestly improved the worst-class coverage over a per-class empirical quantile using the same order-statistic convention. These results position conformal prediction as a coverage control mechanism rather than a universal novelty detector or automation advantage.