Cross-Dataset Intrusion Detection under Benchmark Shift: Directed Transfer, Probability Calibration, and Alert Burden

Machine-learning intrusion detectors commonly report strong performance within one benchmark, but deployment requires transport across traffic environments and realistic target prevalence. This study used three public harmonized NetFlow datasets: NF-UNSW-NB15, NF-ToN-IoT, and NF-BoT-IoT. Their archive attack prevalences were 4.46%, 80.40%, and 97.69%, whereas the balanced analytical samples had 40.23%, 50.00%, and 86.66%. Across the original crossed experiment, mean within-domain AUROC was 0.9773 versus 0.5445 across domains. Removing endpoint-oriented fields reduced uncalibrated cross-domain AUROC from 0.5881 to 0.4990, and source Platt scaling worsened target Brier loss. In a separate repeated evaluation using three uniform 100,000-flow samples at each target’s natural archive prevalence, full-flow models achieved mean AUROC 0.5880, AUPRC 0.6406, F1 0.4348, and Brier loss 0.3398. A threshold fixed at the source-negative 99.5th percentile nevertheless alerted on 36.2% of target flows. Endpoint reduction produced near-chance AUROC (0.4996) and lower F1 (0.3733). High within-benchmark scores therefore did not establish portable ranking, probability quality, or manageable alert rates.