Digitally restoring paintings using convolutional neural networks

Abstract

We digitally restore dirty and varnish-discolored paintings using a convolutional neural network (CNN) trained only on the edges and small sections of the artwork. Our data are images of paintings before and after they were physically treated by a fine art restorer. The training set comes from only 8×8 pixel patches obtained from the edges and small areas within the painting. Instead of RGB, we trained our CNN with patches converted into the La*b* colorspace. We built two CNN models with different inputs: one using the L channel (luminance) and the other using the a*b* channels (chromaticity). This approach allowed us to calculate perceptual color difference and generate virtually cleaned images that closely resemble the actual cleaned paintings. Our results were successful, with a mean color error of ∆E = 5.17 from one painting which is perceptible but acceptable in commercial and industrial contexts. Our findings demonstrate that CNN can infer the appearance of a fully restored painting from just less than 10% of the painting area that has been physically cleaned.