Mitigating hallucination with non-adversarial strategies for image-to-image translation in solar physics

Context. Image-to-image (I2I) translation using generative adversarial networks (GANs) has become a standard approach across numerous scientific domains. In solar physics, GANs have become a popular approach to reconstructing unavailable modalities from physically related modalities that are available at the time of interest. However, the scientific validity of outputs generated by GANs has been largely overlooked thus far. Aims. We address a critical challenge in generative deep learning models, namely, their tendency to produce visually and statistically convincing outputs that might be physically inconsistent with the input data. Methods. We measured the discrepancy between GAN-generated solar images and real observations in two applications: the generation of chromospheric images from photospheric images and the generation of magnetograms from extreme ultraviolet observations. In each case, we considered both global and application-specific performance metrics. Next, we investigated non-adversarial training strategies and network architectures, whose behavior could be adapted to the input at hand. Specifically, we propose an architecture that modulates the generative model’s internal feature maps with input-related information, thereby favoring the transfer of input-output mutual information to the output. Results. Global metrics show that GANs consistently fall short of non-adversarial U-net translation models in physics-constrained applications due to the generation of visually appealing features that do not have any real physical correspondence. Such features are referred to as hallucinations. Additional conditioning procedures carried out via the U-net model, based on the modulation of internal feature maps, can significantly enhance cross-modal image-to-image translation. Conclusions. Our work demonstrates that adaptive instance modulation results in reconstructions that are less prone to hallucinations compared to adversarial settings. An increased robustness to hallucinations is an important advantage in solar physics research where spurious features can be highly problematic.