Digital restoration of paintings using convolutional neural network trained on cleaned edges and local patches

We digitally restore dirty and varnish-discolored paintings using a convolutional neural network (CNN) trained only on the edges and small sections of the artwork. Unlike previous method that artificially simulate varnish aging, our model is trained on real images of paintings before and after the physical restoration by a fine art restorer. Since paintings often undergo planar distortions before treatment, we applied image registration techniques to correct misalignments before processing. We converted the images into the L*a*b* color space to improve perceptual accuracy and trained two CNN models separately: one for luminance (L*) and another for chromaticity (a*b*). This approach enables the CNN to learn structural and color-related features independently, leading to more accurate restoration. The training set comes from only 8×8 pixel patches extracted from the edges and select regions of the painting. This covers less than 10% of the total image area. Our approach successfully predicted the cleaned appearance of entire paintings, achieving a mean color difference of ΔE = 4.14 +/- 2.07 , which is perceptible but within an acceptable range for commercial and industrial applications. Compared to traditional CNN-based restoration in RGB space, our separate processing of luminance and chromaticity yields more perceptually accurate results. Our approach offers a valuable tool for conservators, providing virtual previews of cleaned paintings before physical intervention on the whole painting. It can assist in client consultations, guide conservation efforts, and serve as an alternative when physical cleaning is not feasible.