Enhanced Reflectance: Transformation Imaging for Arts and Humanities

ntroduction
The Digital Humanities Lab (DHLab) is a research group within the faculty of Humanities of the University of Basel. The research profile of the DHLab integrates computer science, digital imaging, computational photography and the accessibility of digital objects in humanities research. The project “Digital Materiality”, an interdisciplinary project in collaboration with the Seminar of Art History, examines how new digital methods and techniques can be used to describe the reflection of light on surfaces of artworks. Of main interest are mosaics and early prints; both categories have a strong interaction with light and standard photographic approaches are not able to capture the dynamic component of the light-surface interdependence that is specific for this kind of objects.

For art historians who study and work with mosaics or any other object of complex surface composition, it is difficult to capture and visualize the important surface features in such a way that research can be done with the reproduction. The major problem is the static nature of photographic reproductions, which does not allow interaction. Static, two-dimensional photographs cannot visualize appropriately the sparkling effect caused by the surface properties of the countless light reflecting tesserae of a mosaic, for example. Similar considerations apply for early prints, books and parchments. The visual impression that these objects convey is hardly delivered by a photograph. Metallic inclusions give the artwork a dynamic appearance caused by the change of reflectance behavior of glossy compared to matte material. Furthermore, the scholar may want to integrate information coming from other types of scientific photographs, such as infrared or ultraviolet illuminated or induced fluorescence images, to increase the visual impression of renderings of artwork. In particular, the combination of such scientific photographs with other imaging techniques like Reflectance Transformation Imaging is advantageous because multiple visual impressions can be combined in a way that would not be possible in reality.

Reflection Transformation Imaging

RTI (Reflectance Transformation Imaging) (Mudge et al, 2007; Malzbender et al, 2001) is a promising approach to go beyond the limitations of conventional photographic methods. However, RTI, as it is used today, has some drawbacks that are critical in the context of the reproduction of mosaics, metallic inclusions or most other artwork.

RTI makes it possible to interactively display the light reflection as a function of the light incident angle and the structure of the surface captured. RTI needs a series of digital images from a fixed camera position, in which the light source is illuminating the object in each capture from a different position. In order to produce RTI renderings of objects of different size, several lighting systems have been developed that consist of a number of light sources that are mounted on the inside of a hemispherical dome. Those light sources can be switched on and synchronized sequentially with a digital camera. The fixed design ensures a fixed and equal distance between the numerous light sources and the object. Thus, with such an automated setup the image capture process is drastically accelerated and, in addition, ensures a high reproducibility. In the Digital Humanities Lab such a dome has been developed as well; In this specific case it is an alloy design, which is equipped with 50 LEDs and an electronic control system, which enables the automatic sequential triggering of the light sources and the synchronization with the camera. The image sequence, recorded in this way, serves in a second step as data basis for a subsequent pixel-based modeling. For this purpose, a mathematical model - normally a function of second order - is fitted for each pixel position, which represents the set of all image points from the different directions of illumination, i.e., it is parameterized so that the square mean error of all data points relative to the function curve becomes minimal.

The reflection model so far used in the original RTI method described by Hewlett Packard (Mudge et al, 2006) corresponds to the mentioned simple second-order function. Thus it has only a relatively low mathematical complexity and therefore a limited precision to represent the actual surface reflection. This specific function, however, describes matt surfaces, a Lambertian radiator, very precisely. However, this method is not suitable for glossy materials, since this simple model does not correspond to the physical laws of gloss reflection. A further disadvantage is that the method, as it is usually used today, needs a specially designed viewer application.

In the Digital Humanities Lab, the method has been enhanced in order to be able to present more complex surfaces that are made from different materials. In our approach the complexity of the mathematical model is increased so that we can handle diffuse and gloss surface components at the same RTI image. In other words, this improvement makes it possible to model and display materials with different gloss levels using the same mathematical model.

A suite for humanities scholars ‘needs

For humanities research another important aspect is the compatibility to web-based Virtual Research Environments (VRE). In principle, a functional graphical "front-end" with a connection to a database can be called a VRE. The aim of such a digital infrastructure for research is to allow scholars to work with methods and tools in the digital domain as they would do it in a conventional “analogue” process. This should be done in such a form that the scientists can intuitively recognize and use the well-known concepts and working methods that are offered by the VRE. This requires some basic functions:

• In any case, a browser-based client-server solution is preferable to a stand-alone application. In such a way collaborative work can be more easily achieved.

• In many humanities disciplines intensive work is being done with image material and objects, in which specific areas (region of interest, RoI) are often to be emphasized. Therefore corresponding graphic elements (lines, polygons) are necessary, with which such areas can be marked. These graphical elements can, for example, be polygonal line sections or rectangles, which allow the marking of object parts.

• The method of evaluating image material, which is common in the human sciences, is of a more qualitative nature. This kind of work requires extensive and powerful tools to capture descriptive and contextual metadata (annotations, transcriptions, comments). These meta-objects have to be linked with the actual primary object and need also to be stored in this way.

The visualization in a VRE must be multi-media. Besides text, image, sound and video, it is also of advantage to be able to display objects in a virtual 3D space.

To be able to integrate an RTI solutions into a webenvironment to support real-time collaborative work needs specific web-technologies (Palma et al, 2010;

MacDonald and Robson, 2010). The presented RTI viewer is fully web-compatible and it can be integrated in most browsers to support high-quality client-side visualization; the key technology that allows this integration is WebGL. This OpenGL-based programming interface, which has been optimized for "embedded systems", is nowadays integrated into any modern Web browsers. WebGL is a license-free standard developed to work seamless in conjunction with the programming language JavaScript. For the application in a browser this means that 3D functionalities are provided without the need to load any additional plug-ins. The performance and range of functions of WebGL are impressive. WebGL is also supported on most mobile devices, such as smartphones and tablets, which further increases the range of applications. The improved performance of WebGL, in contrast to many other graphics libraries, also allows for fluid interactive work. Long rendering times are left out and the visual latency is minimal. WebGL offers a variety of functionalities, ranging from simple grid models to complex animated, textured and illuminated surfaces. The fact that the use of graphic elements for marking RoIs is easily possible with this large range of functions is, of course, self-evident.

A Reflectance Transformation Imaging recording that is represented in a browser using WebGL. Around the left eye of the sacred head is a branding to be recognized, as well as a corresponding commentary, left in the picture window. The parameters of the viewing situation are shown on the right side, which are also stored in the data model. (Source: DHLab, University of Basel)

The integration of all those technologies and new developments allows us to present an improved solution to reproduce and render surfaces of different materials (matt and glossy) in a fully web-based environment implemented in JavaScript and WebGL, running on standard computers and mobile devices. In addition to RTI image processing, photographs in the UV and IR domain can be captured, displayed and superimposed with the same system to allow the user to compare the same region under different light condition. For flexibility, performance and data permanence aspects our RTI image server will follow

the International Image Interoperability Framework

(IIIF). IIIF defines a standardized URL syntax to serve digital images online in the field of cultural heritage and research. The region of interest, resolution, rotation and the file format of a requested image can be indicated on the URL. SIPI, the Simple Image Presentation Interface, developed by DHLab, provides an IIIF compliant image server which is ideally suited for our scope. Due to the fact, that the front-end is compatible with any standard web-browser, it can be integrated in a virtual research environment using Knora. Knora is a software framework, developed by DHLab, for storing, sharing, and working with primary sources and data in the humanities. Knora builds the fundament for the Swiss National Data Center for Research Data in Humanities that is operated by the DHLab. The source code is publicly available on Github at the address in reference.

This integrated environment allows researchers to interactively control the viewing and light conditions of e g. a digital mosaic rendering. Regions of interest can be chosen, annotated, saved, and shared with other scholars. The full set of contextual and technical metadata is stored and time stamped to be fully reloadable and citable back to any point of its history of changes.

Conclusions

The combination of the Enhanced-RTI and the Scientific Image Viewer (SIV) enables us to convey the impressions of these highly dynamic light-surface interactions and the information provided by IR and UV imaging e.g. to researchers who cannot visit the actual artwork in situ. The visual impression can be enriched by meta information that can shared with other scholars. The presented RTI based solution is also helpful to document the current condition of objects more accurately, e.g. before and after a restoration. Sustainability and simplicity of RTI image data is drastically improved by the use of a IIIF server. The presented infrastructure allows the strict separation of image data and meta information. As a result, any RTI image rendering is fully reproducible and therefore perfectly suited for digital archiving, following the requirements of performance, permanence and reliability.

Bibliography

Mudge, M., Ashley, M., and Schroer, C. (2007) A digital future for cultural heritage. Available at:

http://culturalheritageimaging.org/What\ We\ Do/Pu

blications/cipa2007/CIPA\ 2007.Pdf

Malzbender, T., Gelb, D., and Wolters, H. (2001)

Polynomial texture maps. Proceedings of the 28th annual conference on Computer graphics and interactive techniques (pp. 519-528).

Mudge, M., Malzbender, T., Schroer, C., and Lum, M.

(2006). New Reflection Transformation Imaging

Methods for Rock Art and Multiple-Viewpoint Display. In The 7th International Symposium on Virtual Reality,

Archaeology and Cultural Heritage (VAST2006) (pp.

195-200).

Palma, G., Corsini, M., Cignoni, P., Scopigno, R., and Mudge, M. (2010) Dynamic Shading Enhancement for Reflection Trasformation Imaging. ACM Journal on Computing and Cultural Heritage, (pp. 6:1-6:20). (September 2010)

MacDonald, L., and Robson, S. (2010) Polynomial Texture Mapping and 3D Representations. International Archives of Photogrammetry, XXXVIII(8), (pp. 422427).