A novel approach to random and systematic denoising using Gemini AutoEncoder

Abstract Signal denoising is crucial in experimental and industrial applications to ensure data quality and reliable instrument performance. Traditional techniques, such as moving average, moving median, and requency-based filters, as well as deep learning methods like autoencoders, often struggle with systematic or deterministic errors. To address these limitations, this study introduces the Gemini Autoencoder (g-AE), a novel methodology that leverages twin signals representing the same phenomenon but differing only by noise, background and other external phenomena. The g-AE trains the autoencoder to reconstruct one twin from the others, preserving relevant features while effectively removing uncorrelated perturbations. The methodology was validated using synthetic time series with added random and electromagnetic noise, demonstrating superior denoising performance compared to conventional methods and standard autoencoders. Additionally, a novel metric called the Gemini MSE (mean squared error) was introduced to optimize hyperparameter selection, addressing the challenge of deterministic error overfitting when ideal (or target) signals are unavailable. Although developed and tested on time series, the g-AE framework is adaptable to other data types, such as spectra and images. To illustrate its potential, the g-AE was tested in a lidar experimental apparatus for environmental monitoring, where noise removal significantly improved the quality of the information retrieved from measurements. Once trained, the method can be applied without the need for twins, enabling fast, accurate, and real-time compatible denoising.