A Full Factorial Design for the Comparative Evaluation of Intrusion Detection Configurations: Methodology and Controlled Validation

Comparative evaluation of intrusion detection configurations is almost universally reported as a single aggregate score obtained on one dataset. Such a protocol cannot separate the contribution of individual design decisions, cannot detect interactions between them, and provides no estimate of its own measurement error, which is one reason why laboratory results transfer poorly into operation. This paper sets out an experimental protocol that removes these defects, based on a full factorial design with replication. Three factors are varied at two levels each — detector type, threat intelligence enrichment and traffic conditions — in a 2³ layout with five replicates, giving forty runs; the complete treatment is specified, comprising the Cochran test for homogeneity of variances, the Student test for coefficient significance, the Fisher tests for model significance and lack of fit, the Shapiro — Wilk test for residual normality, and the variance inflation factor for multicollinearity. The protocol is validated under controlled conditions: the response is generated from a process model whose coefficients are known in advance, so that the recovered estimates can be compared against ground truth. The procedure reproduced all four generating coefficients within their confidence intervals (4.49 against 4.5; 1.59 against 1.5; −2.97 against −3.0; 0.96 against 1.0), yielded an adequate model (coefficient of determination 0.982; Fisher criterion 247.8 against the tabulated 2.31; lack-of-fit test 0.623 against 2.90), and correctly rejected the three effects that were absent by construction. The protocol detected an interaction of 1.92 percentage points, more than three times the error of the accompanying trend forecast, which indicates the resolution attainable at this design size. Applying the same protocol to traffic captured in an operating network is the subject of further work.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24158880
Primary Topic
Network Security and Intrusion Detection
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A Full Factorial Design for the Comparative Evaluation of Intrusion Detection Configurations: Methodology and Controlled Validation

Dmitriy Khon, Vladislav Kim, Anuar Kadyr, Vladimir Tskhai
Zenodo (CERN European Organization for Nuclear Research)
Network Security and Intrusion Detection
preprint

A Full Factorial Design for the Comparative Evaluation of Intrusion Detection Configurations: Methodology and Controlled Validation

Dmitriy Khon, Vladislav Kim, Anuar Kadyr, Vladimir Tskhai
preprint en

Abstract

Comparative evaluation of intrusion detection configurations is almost universally reported as a single aggregate score obtained on one dataset. Such a protocol cannot separate the contribution of individual design decisions, cannot detect interactions between them, and provides no estimate of its own measurement error, which is one reason why laboratory results transfer poorly into operation. This paper sets out an experimental protocol that removes these defects, based on a full factorial design with replication. Three factors are varied at two levels each — detector type, threat intelligence enrichment and traffic conditions — in a 2³ layout with five replicates, giving forty runs; the complete treatment is specified, comprising the Cochran test for homogeneity of variances, the Student test for coefficient significance, the Fisher tests for model significance and lack of fit, the Shapiro — Wilk test for residual normality, and the variance inflation factor for multicollinearity. The protocol is validated under controlled conditions: the response is generated from a process model whose coefficients are known in advance, so that the recovered estimates can be compared against ground truth. The procedure reproduced all four generating coefficients within their confidence intervals (4.49 against 4.5; 1.59 against 1.5; −2.97 against −3.0; 0.96 against 1.0), yielded an adequate model (coefficient of determination 0.982; Fisher criterion 247.8 against the tabulated 2.31; lack-of-fit test 0.623 against 2.90), and correctly rejected the three effects that were absent by construction. The protocol detected an interaction of 1.92 percentage points, more than three times the error of the accompanying trend forecast, which indicates the resolution attainable at this design size. Applying the same protocol to traffic captured in an operating network is the subject of further work.

Zenodo (CERN European Organization for Nuclear Research)
Satbayev University (KZ)
Network Security and Intrusion Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon