V2IR-CnLDM: a generative visible-to-infrared image translation using ControlNet-guided conditional latent diffusion model

Synthetic data are often cited as a solution to the persistent problem of insufficient amount and diversity of training datasets for data-hungry machine learning models. We present a method of generating physically plausible synthetic thermal infrared (TIR) imagery given a set of input visible RGB images and corresponding descriptive text prompts. Our method is based on a recent neural network architecture, called a ControlNet, which allows adding spatially localized input conditions to a pretrained text-to-image diffusion model via efficient finetuning. We describe our model architecture, implementation, training, and inference details. We then present the results of a task-specific experiment: training a downstream object detector using synthetic TIR data generated with our model. We compare performance between the baseline pre-trained detector and the detector fine-tuned with ground-truth data only and with various hybrid combinations of ground-truth and synthetic data.