Digitally restoring paintings using convolutional neural networks

Authors

  • Mark Jeremy G. Narag ⋅ PH National Institute of Physics, University of the Philippines Diliman
  • Julian Baumgartner ⋅ US Baumgartner Fine Art Restoration, Chicago, Illinois
  • Maricor N. Soriano ⋅ PH National Institute of Physics, University of the Philippines Diliman

Abstract

We digitally restore dirty and varnish-discolored paintings using a convolutional neural network (CNN) trained only on the edges and small sections of the artwork. Our data are images of paintings before and after they were physically treated by a fine art restorer. The training set comes from only 8×8 pixel patches obtained from the edges and small areas within the painting. Instead of RGB, we trained our CNN with patches converted into the La*b* colorspace. We built two CNN models with different inputs: one using the L channel (luminance) and the other using the a*b* channels (chromaticity). This approach allowed us to calculate perceptual color difference and generate virtually cleaned images that closely resemble the actual cleaned paintings. Our results were successful, with a mean color error of ∆E = 5.17 from one painting which is perceptible but acceptable in commercial and industrial contexts. Our findings demonstrate that CNN can infer the appearance of a fully restored painting from just less than 10% of the painting area that has been physically cleaned.

Downloads

Issue

Article ID

SPP-2024-PB-17

Section

Poster Session B (Complex Systems, Computational Physics, and Astrophysics)

Published

2024-06-28

How to Cite

[1]
MJG Narag, J Baumgartner, and MN Soriano, Digitally restoring paintings using convolutional neural networks, Proceedings of the Samahang Pisika ng Pilipinas 42, SPP-2024-PB-17 (2024). URL: https://proceedings.spp-online.org/article/view/SPP-2024-PB-17.