Safety-Aware Weighted Voting for N-version Traffic Sign Recognition System

The N-Version Machine Learning (NVML) system is an approach to improving the reliability of system outputs by employing several different machine learning models in the same system. Voting mechanisms are crucial in NVML systems, influencing the final decision-making process. This paper investigates voting mechanisms for the NVML system by introducing a safety metric based on the Failure Modes and Effects Analysis (FMEA) method. Safety-related weights are assigned to machine learning models in the NVML system to implement weighted voting and weighted soft voting mechanisms. As a case study, we investigate a traffic sign recognition system. Through the FMEA analysis, we categorize the misclassifications of traffic signs based on their severity. The safety metric is defined by the severity with the misclassification probability estimated from the test results and is used for assigning weights to machine learning models. Our experimental results on a real traffic sign dataset show the advantage of safety-aware weighted soft voting in all safety evaluation metrics. Moreover, we use a Large Language Model (LLM) to generate the weights for the voting mechanisms. However, the preliminary results show that the LLM-based approach yields a suboptimal solution compared to our weight assignment method.