Why Benign-Only Log Anomaly Detection Fails Under Drift: Two Mechanisms, Seven Designs, and a Label-Efficiency Bound
Abstract Deep sequence models dominate log-based anomaly detection benchmarks, yet controlled re-evaluations have shown that their reported accuracy collapses under chronological evaluation---without explaining why. We dissect that collapse. Under a leak-free, benign-only protocol on HDFS, BGL, and Thunderbird---chronological and random splits side by side, ten-seed confidence intervals, DeepLog and LogBERT re-run on byte-identical splits---lightweight CPU detectors match or exceed the deep re-runs wherever the template vocabulary is stable (F1 0.92--0.97 at 0.2--0.5\% false-positive rates), and on two of the three benchmarks a one-line unseen-template rule suffices (F1 0.90--0.998), a quantified form of the dataset-triviality critique; HDFS, whose anomalies are structural, is the one system that needs a learned detector. The chronological collapse has two distinct mechanisms. On BGL the template vocabulary is replaced mid-trace: identity evidence is destroyed, the only transferable signal is discriminative fault semantics, and benign-only learning cannot exploit it---six designs without external semantic priors fail concordantly, a frozen pretrained-embedding prior lifts ranking above chance without becoming deployable, and full period-causal supervision yields a ranking (PR-AUC 0.722) whose operating point does not transfer (F1 0.44 causal versus 0.76 oracle). On Thunderbird the signal survives but the calibration era does not: a labeled counterfactual bounds what era-matched recalibration could recover (PCA F1 0.61). Both failures are silent at the alarm level; we identify two label-free self-monitoring statistics that expose them. All code, splits, per-seed results, and re-run artifacts are released.
Authors
- Sarita Nandal (ORCID: https://orcid.org/0000-0002-5548-9582)
Institutions
- Indian Institute of Technology Roorkee (IN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-17
- DOI
- https://doi.org/10.21203/rs.3.rs-11401261/v1
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint