Sim-to-real image to image translation for remote sensing fine-grained ship images using generative diffusion models

Advances in artificial intelligence have enabled the automation of many remote sensing tasks; however, performance remains constrained by dataset quality, especially for fine-grained ship classification in remote sensing imagery, where publicly available datasets suffer from class imbalance and sample scarcity. To address these issues, we propose a novel simulation-to-real style-transfer pipeline for fine-grained ship imagery, comprising three modules: the Simulation Image Generation (SIG) module, the Conditional Image Generation (IG) module, and the Wake Inpainting (WI) module. In the SIG module, we construct an optical remote sensing imaging system capable of producing high-resolution simulated images containing fine-grained ship objects. To overcome the loss of detailed features inherent in global style-transfer methods, the IG module introduces the SPAM-ControlNet algorithm, which generates fine-grained ship images with accurate characteristics. In the WI module, we generate the inpainting region at the stern based on the ship wake model, then apply Stable Diffusion Inpainting to synthesize realistic wake patterns, thereby harmonizing the generated ship objects with the ocean background. This pipeline enables the synthesis of seamless, high-resolution remote sensing images populated with detailed ship objects. Building on this pipeline, we also release a hybrid dataset, FGSCR-SR-12, which combines real and synthetic images across 12 ship classes to mitigate long-tail distribution challenges caused by scarce classes. All code and the FGSCR-SR-12 dataset are publicly available at https://github.com/Slimyer/SPAM-Controlnet.