A Data-Driven Study of IoT Malware Classification Models: Insights Into Temporal, Architectural, and Spatial Inconsistency Challenges

To combat the growing IoT malware threat, many studies propose ML-based classification solutions, but the lack of comprehensive evaluations limits insights for developing new solutions and selecting models. Given this necessity, this work evaluates IoT malware classification models under three key challenges: temporal, architectural, and spatial inconsistencies between development and deployment datasets, which can be regarded as variables characterizing the dataset, and the challenges arise from the variable values inconsistency between the two stages. To improve the conclusions’ comprehensiveness, effectiveness, and generalizability, the evaluation is organized hierarchically across three levels based on model generation, sample variation, and inconsistency assumptions. Given the complexity of the model development pipeline, our evaluation treats each model individually and aims to conclude impacts across all models. The analysis reveals that temporal and architectural inconsistencies significantly degrade model performance, with architectural inconsistency having a greater impact, despite cross-architecture designs. Temporal inconsistency effects vary with temporal value differences, while spatial inconsistency has minimal impact, even with substantial spatial variation. Furthermore, we use one-way ANOVA to identify features contributing to family distinguishability, temporal stability, and architectural generalizability that benefit future solution design. Meanwhile, we studied a specific example to study the model performance degradation under architectural inconsistency. Finally, we summarize the lessons learned and outline potential research directions to address these challenges.