Multi-Task Learning for Traffic Sign Recognition using Multi-Scale Convolutional Neural Networks

Traffic signs are an essential component of road infrastructure. According to the Department of Transportation, Indonesia has over 300 distinct traffic signs, categorized based on their functions and purposes. TSR systems have been widely integrated into various intelligent transportation technologies, such as Driver Assistance Systems (DAS), Advanced Driver Assistance Systems (ADAS), and Autonomous Driving Systems (ADS). The output generated by TSR serves as a critical input for DAS, ADAS, ADS, and other intelligent systems. This article presents a CNN-based classification for traffic sign recognition using multi-task learning (MTL), focusing on traffic signs in Indonesia. The dataset was collected from direct capture with the help of a cellphone camera, indirect capture by utilizing screenshots on a digital map application, and they are captured from several different angles, during the day and at night. The proposed CNN architecture incorporates multi-scale within an MTL framework. The use of a multi-scale approach will hopefully enhance the model’s ability to recognize traffic signs in varied and complex environments. And the integration of MTL will enable the model to handle multiple related tasks concurrently, sharing learned features across tasks. During the training stage, the MS-CNN outperformed a standard CNN model by demonstrating lower initial loss, higher starting accuracy, and achieving 100% accuracy by the 8th epoch with a minimal error rate of just 0.003. In the testing stage, the model achieved exceptional results, as shown by the confusion matrix, it successfully classified all traffic sign types (10 classes) and accurately categorized each sign into one of two categories—warning or prohibition. All performance metrics, including precision, recall, and F1-score, reached 100% for both output tasks, confirming the robustness and reliability of the model.