Synthetically Generated IR Image Data for Training Deep Learning Models and Application to Object Detection

A major challenge in machine learning is the need for extensively annotated training data to ensure model robustness, particularly in rare and unpredictable scenarios. Autonomous vehicles (AVs) rely on object detection models that must function effectively even in extreme conditions. These object detection models require large amounts of image data for effective training. Building on a previously developed and published open-source, physics-based infrared (IR) sensor model in Unreal Engine, this research focuses on generating synthetic IR image datasets that are automatically annotated using a new addition of an auto-annotation tool for the model, and validating the effectiveness of synthetic IR data compared to real-world IR datasets in reducing the effort and cost associated with data collection. The performance of a YOLOv8 object detection model trained on synthetic IR images is compared against a model trained on real IR datasets from FLIR, which contains 25,000 annotated images. The motivation for using synthetic training data is threefold: (1)it significantly reduces the cost and effort required to collect diverse real-world IR datasets by replacing expensive sensor setups with virtual equivalents, (2) it enables auto-annotation, eliminating the labor-intensive manual labeling required for real images, and (3) it allows simulation of complex or rare scenarios that would be difficult or impossible to capture in real-world conditions. Our approach models a physics-accurate IR sensor using post-process materials in Unreal Engine, simulating a microbolometer-based detection mechanism. The generated synthetic IR images are automatically annotated by projecting 3D object bounding boxes into 2D screen space. We evaluate four different dataset compositions: (a) 100% synthetic IR images, (b) 95% synthetic + 5% real IR, (c) 80% synthetic + 20% real IR, and (d) 100<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup> real IR images. Model performance is analyzed using precision-recall curves, mean Average Precision (mAP), and confidence-based evaluation. Results indicate that models trained on a mix of synthetic and real data achieved comparable detection performance to those trained on real data, with acceptable confidence levels for warm-blooded objects (pedestrians, animals). However, confidence levels decline when detecting vehicles due to their uneven heat distribution, highlighting a limitation in current heat transfer modeling. These findings suggest that synthetic datasets, particularly when mixed with real data, can significantly reduce the dependence on costly real-world datasets while maintaining effective model performance.