Protocol-Constrained Adversarial Evaluation of MQTT-Based Intrusion Detection Systems: Quantifying the Realism Gap

Abstract: Machine Learning Intrusion Detection Systems (ML IDS) are vulnerable to adversarial attacks—intentional perturbations that cause misclassification. Most studies generate unconstrained feature-space attacks that ignore network-protocol rules, creating a realism gap in resource-constrained Internet of Things (IoT) networks where high theoretical evasion rates may not reflect a genuinely deployable threat. Following a Design Science Research methodology, this study develops and empirically evaluates a protocol-aware adversarial evaluation framework for Message Queuing Telemetry Transport (MQTT)-based IDS. A Random Forest baseline classifier was trained on the publicly available MQTTEEB-D dataset (n = 222,813). A three-layer protocol constraint model (field, temporal, and state-machine rules) was formalized, with field and temporal constraints encoded directly into the fitness objective of an evolutionary attack generator and benchmarked against two unconstrained feature-space attacks—Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD)—using Evasion Success Rate (ESR) and Protocol Validity Rate (PVR). The baseline classifier achieved 91% accuracy and a macro F1 score of 0.86. While PGD reached 24.16% ESR, it yielded 0% PVR; in contrast, the protocol-aware evolutionary attack achieved 6.00% ESR while maintaining 71.3% PVR (McNemar's p = 8.30 × 10⁻¹⁷, Cohen's h = 0.533). These findings demonstrate that traditional feature-space attacks overestimate deployable risk, whereas protocol-compliant attacks reveal a smaller yet statistically significant real-world vulnerability. Keywords: Adversarial Machine Learning; Intrusion Detection Systems; MQTT Protocol; Internet of Things Security; Protocol-Aware Perturbation

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24311252
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Protocol-Constrained Adversarial Evaluation of MQTT-Based Intrusion Detection Systems: Quantifying the Realism Gap

Elisha Odira Abade, Benjamin Isaac Macharia Kinyanjui
1 links
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

Protocol-Constrained Adversarial Evaluation of MQTT-Based Intrusion Detection Systems: Quantifying the Realism Gap

Elisha Odira Abade, Benjamin Isaac Macharia Kinyanjui
preprint en

Abstract

Abstract: Machine Learning Intrusion Detection Systems (ML IDS) are vulnerable to adversarial attacks—intentional perturbations that cause misclassification. Most studies generate unconstrained feature-space attacks that ignore network-protocol rules, creating a realism gap in resource-constrained Internet of Things (IoT) networks where high theoretical evasion rates may not reflect a genuinely deployable threat. Following a Design Science Research methodology, this study develops and empirically evaluates a protocol-aware adversarial evaluation framework for Message Queuing Telemetry Transport (MQTT)-based IDS. A Random Forest baseline classifier was trained on the publicly available MQTTEEB-D dataset (n = 222,813). A three-layer protocol constraint model (field, temporal, and state-machine rules) was formalized, with field and temporal constraints encoded directly into the fitness objective of an evolutionary attack generator and benchmarked against two unconstrained feature-space attacks—Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD)—using Evasion Success Rate (ESR) and Protocol Validity Rate (PVR). The baseline classifier achieved 91% accuracy and a macro F1 score of 0.86. While PGD reached 24.16% ESR, it yielded 0% PVR; in contrast, the protocol-aware evolutionary attack achieved 6.00% ESR while maintaining 71.3% PVR (McNemar's p = 8.30 × 10⁻¹⁷, Cohen's h = 0.533). These findings demonstrate that traditional feature-space attacks overestimate deployable risk, whereas protocol-compliant attacks reveal a smaller yet statistically significant real-world vulnerability. Keywords: Adversarial Machine Learning; Intrusion Detection Systems; MQTT Protocol; Internet of Things Security; Protocol-Aware Perturbation

Zenodo (CERN European Organization for Nuclear Research)
University of Nairobi (KE)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon