Beyond Performance Metrics: The Critical Role of Resource-Based Evaluation in Assessing IoT Attack Detectors

The proliferation of threats within the Internet of Things (IoT) environment is intensifying, largely due to the inherent limitations of this technology. The panoply of anti-threats based on artificial intelligence suffer from the complete embedment of models in limited resources. Tiny Machine Learning (TinyML) is presented as an opportunity in optimizing and selecting machine learning algorithms specifically tailored for intrusion detection systems (IDS) on limited-resource devices. This article addresses the challenges that must be overcome to enable the deployment of machine learning models on devices with constrained resources. In particular, it introduces additional indicators that could influence the algorithmic design of IoT models. Utilizing the PyCaret tool on the TON_IoT dataset, which encompasses nine distinct attacks, we developed and evaluated our approach for selecting the optimal algorithm from fourteen supervised learning models. The proposed tool, beyond the traditional six performance metrics, emphasizes resource consumption metrics, including memory, processor usage, battery life, and execution time – key considerations for TinyML in model refinement and selection. This study has identified less resource-intensive models suitable for developers in the design of IDS for IoT systems. We believe this research offers a foundational framework for the development of lightweight and efficient IoT vulnerability detection solutions.