Evaluating Protocol-Aware Testing Strategies for Cloud-Based MQTT Systems : A Software Engineering Evaluation of Reliability, Robustness, and Resilience in Smart Home IoT Systems

Background. Smart home energy management systems (HEMS) rely heavily on reliable machine-to-machine communication to coordinate distributed devices such as thermostats and lighting units. MQTT has emerged as one of the most widely adopted communication protocols for IoT systems due to its lightweight publish/sub- scribe model. However, MQTT-based control systems remain vulnerable to protocol- level defects, message inconsistencies, and network disturbances, which can degrade reliability and system resilience. Traditional functional testing approaches often fail to adequately capture these communication-level failures, particularly under real- world network disruptions. Objectives. The objective of this thesis is to investigate how protocol-aware au- tomated testing practices—specifically contract-based schema validation, property- based testing, and fault and chaos injection can be applied as software engineering techniques to improve key software quality attributes of MQTT-based smart home control systems. These attributes include reliability, robustness, interface correct- ness, and architectural compatibility between distributed components. The thesis aims to quantitatively assess how these testing practices influence defect rates, sys- tem behavior under failure conditions, and performance overhead when integrated into automated test and continuous integration pipelines. Methods. A minimal Home Energy Management System (HEMS) control path was implemented using an MQTT broker and simulated smart home devices. Four experimental configurations were evaluated: (i) baseline testing, (ii) contract-based testing, (iii) property-based testing, and (iv) fault and chaos injection. Controlled message workloads and injected failure scenarios were applied where applicable. The evaluated metrics included defect rate per 10,000 messages, message latency per- centiles, throughput, message loss and duplication rate, and mean time to recovery (MTTR). All experiments were conducted using a reproducible Docker-based test infrastructure. Results. The results show that contract-based testing reduced protocol-level defects by up to 63.8% compared to baseline testing, with a moderate increase in message latency of approximately 11%. Property-based testing uncovered numerous robust- ness issues related to boundary values and unexpected message sequences that were not detected by schema validation alone. fault and chaos injection exposed recovery behavior under broker and network failures, enabling quantitative analysis of system resilience and MTTR. Conclusions. The findings demonstrate that protocol-aware utomated testing sub- stantially improves the reliability and fault tolerance of MQTT-based smart home control systems while maintaining acceptable runtime overhead. Combining contract validation, property-based testing, and fault injection provides a systematic software engineering approach for strengthening communication robustness in message-driven IoT systems.