Enhancing Capability and Reliability of IoT Data Anomaly Detection through the Integration of Lambda Architecture and MQTT Protocol

The detection process is often hampered by the limitations of edge devices, high latency of sending messages to the cloud, message loss due to burst traffic, and decreased model accuracy due to concept drift. The main objective of this research is to improve the accuracy and speed of IoT anomaly detection through a combination of speed layers and batch layers in an MQTT-based Lambda architecture, reducing end-to-end latency, suppressing the detection error rate, and testing the system's reliability under high data traffic conditions and when concept drift occurs. The type of research used is experimental research with a quasi-experimental approach. The research subjects involved data streams from approximately 40 IoT devices. The data collection method was carried out using an MQTT broker (Mosquitto/EMQX) connected to a stream processor for the speed layer. The research instruments included system logs, latency tracers, throughput meters, and a confusion matrix. Data analysis was done through preprocessing, anomaly detection modeling, and a performance comparison between the batch-only baseline and Lambda integration. The results show that integrating the Lambda architecture with the MQTT protocol can significantly improve the performance of IoT anomaly detection. The F1 score increased from 0.81 to 0.90, the end-to-end latency decreased from 1.8 seconds to 0.35 seconds, and the false positive rate decreased by 32% compared to the batch-only method. The system also proved more reliable under high data loads, with less than 0.2% message loss when using quality of service 1. Furthermore, the six-hour, despite concept drift training strategy successfully maintained detection accuracy at F1 ≥ 0.88 despite concept.