Scenario-Wise Comparative Evaluation of Lightweight YOLO Models for Flame and Smoke Detection

Flames and smoke vary in appearance across fire environments, limiting the ability of aggregate detection metrics to characterize model applicability. This study compared lightweight models from ten YOLO versions using an integrated dataset comprising five fire scenarios. Basic image statistics and DINOv2 feature embeddings characterized the scenario groups, while class- and scenario-specific evaluation examined performance differences obscured by aggregate metrics. All models used the same fixed data splits. Aggregate test results were summarized as means and sample standard deviations across three training runs using seeds 0, 21, and 42. The evaluation covered detection accuracy, object-size-specific average precision, inference speed, and fixed-threshold error characteristics. No single model achieved the highest class-specific AP50-95 across all scenarios. Wildland and outdoor fires yielded AP50-95 values of 0.285–0.358 for flame and 0.293–0.392 for smoke, whereas smoke AP50-95 exceeded 0.90 for all evaluated models in electrical equipment and industrial facility fires and in road and vehicle fires. Small objects also remained challenging across models. These findings demonstrate how scenario-stratified evaluation reveals detection weaknesses beyond aggregate rankings and supports baseline selection according to the target environment, object class, and computational requirements.