Multimodal Road Surface Condition Classification - Integrating Contextual Features into Vision Transformer Models

Road surface condition classification is important for traffic safety and winter road maintenance. CCTV images provide direct visual information about the road surface, but image based models can be affected by low illumination, blur, reflections, and visually ambiguous conditions. This thesis investigates whether structured metadata can improve road surface condition classification when combined with visual features from a Vision Transformer. Thus, four models are developed and evaluated: Metadata Only, Image Only, Concatenation Fusion, and Cross Attention Fusion. The task is formulated as a three class classification problem with the classes dry, wet, and snow. The Image Only model uses a pretrained Vision Transformer to extract visual features from CCTV images, while the Metadata Only model uses structured station level and environmental metadata. The two fusion models combine visual features with metadata embeddings using either simple concatenation or a cross attention based fusion mechanism. The models are evaluated on an external evaluation set using accuracy, Macro F1, and confidence based accepted sample metrics. The results show that metadata alone is not sufficient for reliable road surface condition classification in the final external evaluation. However, both fusion models improve over the Image Only baseline, showing that metadata provides useful complementary information when combined with visual features. Concat Fusion achieves the highest overall accuracy and coverage, while Cross Attention Fusion achieves the highest Macro F1 and the strongest accepted sample performance. This indicates that simple fusion is effective for improving overall performance, while cross attention may provide better class balanced and high confidence predictions. Additional analyses show that model performance varies across metadata groups, illumination conditions, image resolutions, countries, stations, and classes. Weather related metadata provides the strongest standalone contribution, while static metadata groups require careful interpretation because they may encode station or region specific patterns. The results also show that night images, station specific characteristics, and dry to wet confusion remain important challenges. Overall, this thesis shows that contextual metadata can improve road surface condition classification when used together with CCTV images. However, the benefit depends on the fusion strategy, metadata quality, data distribution, and evaluation setting.