While admiring a work of art – possibly damaged by the passage of time, the elements, human intervention or wear-and-tear – we have all played at least once the part of an art restorer and interpreter. “What did the art piece look like when it was created?” “What can we see and not see in it now?” “What materials were used to make it and have they changed over time?” are only few of the questions we may have asked ourselves. How to answer them in an informed while objective manner? Surprisingly maybe, this is where mathematics can play a role.
“What did the art piece look like when it was created?” “What can we see and not see in it now?” “What materials were used to make it and have they changed over time?”
Mathematical methods for the analysis and the processing of artworks have become very popular over the last decade thanks to the emergence of digitization in the arts. Notably, digital restoration problems have attracted the attention of the mathematical imaging community which has started applying state-of-the-art image restoration methods to enhance certain aspects in art pieces and help answering the questions above.
Digital restoration of Illuminated Manuscripts by ‘inpainting’
One aspect of our work is the application of digital image restoration methods to illuminated manuscripts.
Illuminated manuscripts constitute the largest and best preserved repository of European paintings before 1500. Although they survive in far larger numbers and better condition than any other form of medieval painting, some have suffered from the natural degradation of pigments over the centuries as well as from iconoclasm and censorship which have altered their original images. Due to the very thin and delicate layers of their pigments, illuminations are not normally restored by conservators in the way that panel and wall paintings often are.
Illuminated manuscripts constitute the largest and best preserved repository of European paintings before 1500.
Virtual restoration and virtual manipulation is often the only way to recover damaged illuminations. The mathematical technique used for ‘painting in’ the missing contents of a digital image using the available information is called image inpainting.
In our work, we applied image inpainting techniques to restore contents in the illuminated manuscripts which have gone missing due to large and irregular scratches on the gold leaves. To detect the scratches, we propose a simple method which only requires a single input by an art expert. Such input is used as a seed for the identification of the surrounding image region which shares with the given seed the same colour features. By means of standard clustering algorithms, such “training” examples can be used to find in the image all the similar damaged areas in an automatic way.
Once all such areas are identified, a mathematical “copy & paste” inpainting algorithm can be applied to scan within the whole image the most plausible piece of information to be transferred into the damaged regions so as to reveal how the illumination might have looked before the damages occured.
In this example the image content in the damaged areas of the illumination is completely lost and it was estimated only from the information available in the rest of the picture.
How to reconstruct ‘overpainted’ illuminations
This, however, is not the only kind of degradation encountered in the process of restoration of illuminated manuscripts. In some cases parts of an illumination may have been painted over. A prominent example of such a case, and how art historical and scientific analysis and mathematical image restoration could help unveiling what is invisible to the human eye is an illumination in Claude of France’s Primer featured in a 2016 article in the Guardian, and exhibited in the 2016 Colour exhibition at the Fitzwilliam Museum.
The process began with a digital colour photograph of an overpainted scene and an infrared image of it. The infrared image revealed the original structure of the painting beneath the later additions. The digital restoration based on the colour photograph and the infrared image involved various steps. First we marked the part that we would restore and created a mask. Then, we solved on the mask so-called osmosis filtering, a process modelled by a so-called partial differential equation. Osmosis takes the colour given on the boundary of the mask and propagates it into the mask following the structure extracted from the infrared image.
Just like stepping inside a painting
In addition to digital restoration, the digitalization of artwork (in conjunction with virtual reality and related technologies) has made it possible to create 3D or animated versions of artwork that can only be experienced digitally (see for example here, here, here, and here). In this part of our work we focused on the former, applying 3D conversion techniques originally developed for Hollywood films and applying them instead to illuminated manuscripts and paintings.
Given an input image, the term “3D conversion” refers to the generation of an output of left and right eye pair of images which can be viewed using 3D glasses. The approach we took required us, among other things, to build a plausible 3D model of the scene depicted in the painting, onto which the painting itself can be projected. Other steps, which we illustrate in the animated gif above, include rendering the 3D scene from a new viewpoint and inpainting previously occluded areas.
In most cases, this process was straightforward. However, in the case of Edvard Munch’s “The Scream” we discovered something interesting. The screaming person that is the focus of “The Scream” partially occludes the railing of a bridge. When performing the “disocclusion” step of the 3D conversion process we discovered that the two sides of the bridge – if extrapolated into the region occluded by the person – do not meet! As a result, we had to introduce a “kink” into the 3D model of the bridge, as below.
The last 50 years have seen an impressive development of mathematical methods for the analysis and processing of digital images, mostly in the context of photography, biomedical imaging and various forms of engineering. The arts have been mostly overlooked in this process, apart from a few exceptional works in the last 10 years. With the rapid emergence of digitisation in the arts, however, the arts domain is becoming increasingly receptive to digital image processing methods and the importance of paying attention to this therefore increases. Mathematical methods provide a powerful toolkit for digital art restoration and manipulation, and opening up new ways for viewing, analysing and interpreting works of art.
An overview of the 3D conversion process is illustrated in this video.
More visualizations of some of our 3D converted paintings can be found here, here, here, and here.