Prioritizing Vulnerability Assessment Items for IoT Devices Based on Suitability Evaluation Using LLMs

With the rapid increase in demand for IoT devices, malicious attacks targeting vulnerabilities in IoT devices have been frequent in recent years. It is highly expected that the vulnerabilities can be removed from them through vulnerability assessment. However, the wide variety of IoT devices is not standardized, and it is difficult to set up vulnerability assessment items mechanically for those IoT devices, which causes a major obstacle to automate the vulnerability assessment for IoT devices. In this paper, we propose a method to prioritize vulnerability assessment items for every IoT device by effectively utilizing large language models (LLMs). The proposed method generates the answers that take into account the specifications of individual IoT devices using an LLM by introducing Retrieval Augmented Generation (RAG), and determines how much suitable each vulnerability assessment item is for every IoT device by calculating the suitability using semantic entropy. At that time, the proposed method introduces hybrid search with reranking as a search method for related chunks in RAG. Through binary classification of vulnerability assessment items, the average area under the curve (AUC) of 0.753 was achieved for five IoT devices. We confirmed that the proposed method is more effective in evaluating the suitability of the items to the target device specifications than the methods using keyword search, vector search, and hybrid search with RRF (Reciprocal Rank Fusion).