Why Benign-Only Log Anomaly Detection Fails Under Drift: Two Mechanisms, Seven Designs, and a Label-Efficiency Bound

Abstract Deep sequence models dominate log-based anomaly detection benchmarks, yet controlled re-evaluations have shown that their reported accuracy collapses under chronological evaluation---without explaining why. We dissect that collapse. Under a leak-free, benign-only protocol on HDFS, BGL, and Thunderbird---chronological and random splits side by side, ten-seed confidence intervals, DeepLog and LogBERT re-run on byte-identical splits---lightweight CPU detectors match or exceed the deep re-runs wherever the template vocabulary is stable (F1 0.92--0.97 at 0.2--0.5\% false-positive rates), and on two of the three benchmarks a one-line unseen-template rule suffices (F1 0.90--0.998), a quantified form of the dataset-triviality critique; HDFS, whose anomalies are structural, is the one system that needs a learned detector. The chronological collapse has two distinct mechanisms. On BGL the template vocabulary is replaced mid-trace: identity evidence is destroyed, the only transferable signal is discriminative fault semantics, and benign-only learning cannot exploit it---six designs without external semantic priors fail concordantly, a frozen pretrained-embedding prior lifts ranking above chance without becoming deployable, and full period-causal supervision yields a ranking (PR-AUC 0.722) whose operating point does not transfer (F1 0.44 causal versus 0.76 oracle). On Thunderbird the signal survives but the calibration era does not: a labeled counterfactual bounds what era-matched recalibration could recover (PCA F1 0.61). Both failures are silent at the alarm level; we identify two label-free self-monitoring statistics that expose them. All code, splits, per-seed results, and re-run artifacts are released.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11401261/v1
Primary Topic
Software System Performance and Reliability
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Why Benign-Only Log Anomaly Detection Fails Under Drift: Two Mechanisms, Seven Designs, and a Label-Efficiency Bound

Sarita Nandal
4 links
Research Square
Software System Performance and Reliability
preprint

Why Benign-Only Log Anomaly Detection Fails Under Drift: Two Mechanisms, Seven Designs, and a Label-Efficiency Bound

Sarita Nandal
preprint en

Abstract

Abstract Deep sequence models dominate log-based anomaly detection benchmarks, yet controlled re-evaluations have shown that their reported accuracy collapses under chronological evaluation---without explaining why. We dissect that collapse. Under a leak-free, benign-only protocol on HDFS, BGL, and Thunderbird---chronological and random splits side by side, ten-seed confidence intervals, DeepLog and LogBERT re-run on byte-identical splits---lightweight CPU detectors match or exceed the deep re-runs wherever the template vocabulary is stable (F1 0.92--0.97 at 0.2--0.5\% false-positive rates), and on two of the three benchmarks a one-line unseen-template rule suffices (F1 0.90--0.998), a quantified form of the dataset-triviality critique; HDFS, whose anomalies are structural, is the one system that needs a learned detector. The chronological collapse has two distinct mechanisms. On BGL the template vocabulary is replaced mid-trace: identity evidence is destroyed, the only transferable signal is discriminative fault semantics, and benign-only learning cannot exploit it---six designs without external semantic priors fail concordantly, a frozen pretrained-embedding prior lifts ranking above chance without becoming deployable, and full period-causal supervision yields a ranking (PR-AUC 0.722) whose operating point does not transfer (F1 0.44 causal versus 0.76 oracle). On Thunderbird the signal survives but the calibration era does not: a labeled counterfactual bounds what era-matched recalibration could recover (PCA F1 0.61). Both failures are silent at the alarm level; we identify two label-free self-monitoring statistics that expose them. All code, splits, per-seed results, and re-run artifacts are released.

Research Square
Indian Institute of Technology Roorkee (IN)
Software System Performance and Reliability
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon