Limits of Web Attack Detection in Real-World Production Environments

Machine learning (ML) detectors of web and application programming interface (API) attacks report near-perfect laboratory performance but overwhelm analysts with false alerts in production. We argue analytically and support experimentally that when the attack class is produced by a fixed set of tools on a testbed, the learned decision function approximates the complement of the training benign distribution, so its false alert rate on real traffic depends mainly on benign distribution shift. We formalize this as a coverage condition with a false alert bound and measure coverage directly. On four real-traffic datasets, a text-only character-level detector trained on the target domain’s own benign traffic yields alert rates of 0.42% and 0.61% on two independent domains, against 99.5% after training on a different source, with recall of at least 93% on independent attack datasets, although residual source confounding cannot be excluded. The detector outperforms a declared regular-expression baseline on real attacks, with 100% recall against 45.8%, and the entire difference arises from attacks outside the rule dictionary. Attack family attribution does not transfer. Target domains lack request bodies, so results hold for the uniform resource identifier (URI) and header channel. Testbed labels derive from provenance. A public artifact reproduces every result.