MemoryGraph: Digital Critique of Old Photographs Using a Mobile App that Enhances the Interpretation of Landscape

Introduction

MemoryGraph is a medium to record memories as the augmentation of a photograph, which is a medium to record photons. MemoryGraph is a new photographic technique that creates a layer of “memories” (or more precisely, time-series photographs) which are taken by the same composition at different times. Photographs of the same composition can be taken by traditional cameras, but this requires substantial effort, because matching a printed photograph with the current landscape requires mental rotation (Shepard et al., 1971), a psychologically demanding task. MemoryGraph simplifies this task through direct overlay of a photograph and the current landscape on a camera viewfinder with adjustable transparency. We demonstrate that this simple idea opens up new possibilities for the critical interpretation of photographs in the context of digital critique.

Related Methods

MemoryGraph focuses on the spatio-temporal relationship between photographs and the real world. This field of interest has given rise to many methods similar to MemoryGraph. For this reason, we begin with a comparison between MemoryGraph and these related methods in order to characterize MemoryGraph's unique properties.

First, it is necessary to compare MemoryGraph with time-lapse animation, which is also a time-series image of the same composition. The fundamental difference between time-lapse animation and MemoryGraph is in time scale and device dependence. Time lapse animation deals with high frequency observations of seconds to minutes using the same location-fized camera, while MemoryGraph deals with low frequency observations over a period of days to years. These observations may potentially use different cameras that are not fixed at the location. In short, MemoryGraph is a tool to realize fixed point observation at any place for any time interval.

Secondly, we contrast MemoryGraph with augmented reality (AR), which focuses on the alignment of the real space and the virtual space so that a photograph can be seen as an overlay on the real space through a camera viewfinder. On the contrary, MemoryGraph focuses on the overlay of a photograph on a camera viewfinder as a graphic reference to illustrate the composition of a further action (namely, taking a photograph). This key contrast suggests fundamentally different roles between augmented reality and MemoryGraph. Augmented reality is a tool for exhibition, in the sense that alignment is controlled by a tool, while the user is allowed to be a passive visitor who experiences an environment prepared by someone else. Quite the opposite, MemoryGraph is a tool for participation: the “visitor” controls the alignment, and the user should be an active explorer who searches for the best match between photograph and real-life landscape. In short, MemoryGraph is a tool for actions, such as participatory annotation.

Our final comparison discusses photo-sharing services dedicated to old photographs. For example, Historypin (Shift, 2010) is designed to share photographs of the past using a map interface, while an advanced system might use a street-view interface to place photographs in 3D. Both tools aim to link photographs to the real world, however, their methods of achieving this goal do not rely on ventures into the “real world” to “place” the photographs. Photo sharing services depend on “arm-chair” annotators, but MemoryGraph depends on “field” annotators who visit a real place to take another photograph. In short, MemoryGraph is a tool to motivate people to move in the real world.

Proposed Method

MemoryGraph is designed as a mobile app for two reasons. First, the idea of “graphic reference” overlay on a camera viewfinder cannot be implemented on traditional cameras that do not expose API (application programming interface). By re-purposing the viewfinder of a smart phone, MemoryGraph can extend the grid-based reference of a traditional camera viewfinder to a graphic reference such as an old photograph. Secondly, in order employ MemoryGraph as a field work tool, the use of a mobile phone is the best choice for mobility and also for real-time information sharing with the server.

The task of the user is to find the best match between a graphic reference and the real world using an opaque viewfinder with adjustable transparency. Direct comparison between two scenes not only reduces the burden of mental rotation for the user, but also makes the search enjoyable: the movement of the user gives real-time visual feedback that suggests how “good” the move is. This gamification effect motivates users to search for better matching, and promotes photographic crowd-sourcing for participatory annotation.

The outcome of this task is two types of data. One is a photograph that records the current landscape, and another is the metadata of the photograph such as latitude, longitude, and direction observed by sensors in a mobile phone. Metadata may be enhanced later to add a title, and description (among other things), and users can upload metadata and photographs to the server for sharing. The uploaded data can then be used for scholarly research, because the GPS coordinates and the temporally different views of the landscape are valuable resources for understanding the landscape changes.

Results

The MemoryGraph (CODH, 2016) app is available for free on Google Play, but an iOS version has not yet been released. The predecessor app, MemoryHunt (Kitamoto, 2015), has been used for both the study of cultural landscapes and the monitoring of disaster recovery (DSR, 2014).

To study cultural landscapes, we held several workshops with both scholars and laypersons who tried the app in the field. An example is shown in Figure 1, where the reference image was a photograph of Imperial Palace Moat in Tokyo. The photograph was easy to interpret, but the actual place was difficult to find. Figure 2 shows how participants walked along the moat

to find the best match. At each place, participants took the photograph that they believed to be the best match, but the best solution of Figure 1 was found after many trials of all the participants.

Figure 1. Comparison of two photographs of the same composition. “A view from Miyakezaka to the ministry of justice” taken around 1911 and the current landscape. Courtesy of National Diet Library.

Figure 2. Participants walked along the moat to find the best match, and their trials are marked by icons. The final solution of Figure 1 was obtained at the north end of the moat.

For the second purpose, we held workshops at Kobe in Japan, which was severely damaged by the earthquake in 1995, and Aceh in Indonesia, which was severely damaged by the tsunami in 2004, to understand how the city recovered from the disaster. The app was enjoyed by local people in two countries, and some children were involved more actively than adults due to the gamification effect.

MemoryGraph can be generalized in several ways. First, it can be generalized from landscape to object: for example, time-series photographs of the same person at different places. Second, it can be generalized to cross-media reference: for example, taking the same composition at a “sacred place” of pop-culture work inspired by the real landscape.

Discussion on Digital Critique

Digital critique, or what has been referred to in other words as digital criticism (Kitamoto, 2016) or data criticism (Kitamoto, 2014), was proposed by the author as a framework for digital scholarship in the humanities. It has been applied to the criticism of non-textural sources such as maps and photographs with the intent of using them as evidence for historical studies. Although the digital humanities are often considered quantitative or metric humanities, digital critique focuses on the digital creation and management of humanities knowledge. Several types of digital tools have been designed for the critical interpretation of textual and non-textural sources, including success stories such as the identification of Silk Road ruins, or the characterization of mistakes in the renovation of old Beijing maps (Kitamoto et al., 2014) (Kitamoto et al., 2016). We claim that MemoryGraph is another digital critique tool that asks the following research questions in order to use a photographic source as historical evidence.

The first research question asks what the variance and invariance is in the target landscape over time. Invariance is often found as artificial features such as roads and monuments, or natural features such as a mountain ridge, but in other cases, the identification of invariance requires training of the user in terms of interpretive performance on the historical landscape.

Secondly, we ask how historical evidences can be best integrated across different sources, such as old maps, old photographs and historical databases. Invariance found through the app is a hint to links different sources over time, and this contributes to expanding our understanding by integrating knowledge from different sources.

Our final research question asks why the photograph was taken in this setting. This is because the best match is theoretically achieved by the same physical posture between the original photographer and the user (Kitamoto, 2015). This means that as a user tries to find the best match using the app, a user can re-experience the original photographer who took the photograph in the same posture. The physical overlay of the user and the photographer may lead to research questions investigating the emotion or the way of thinking of the original photographer at the time. This has potential to give birth to new interpretation of photographs from the physical or emotional perspective.

Acknowledgments

The project was partially supported by JSPS

Kakenhi Center Grant Number 16H02920 and by Center for Integrated Area Studies (CIAS), Kyoto University. The core contributor to the app is Mr. Tomohiro Ikezaki.

Bibliography

Center for Open Data in the Humanities (CODH)

(2016). “MemoryGraph.” https://mp.ex.nii.ac.jp/mg/, (accessed on April 6, 2017).

Digital Silk Road Project (DSR) (2014). “MemoryHunt.” http://dsr.nii.ac.jp/memory-

hunting/, (accessed on April 6, 2017).

Kitamoto, A., and Nishimura, Y. (2016). "Digital criticism platform for evidence-based digital humanities with applications to historical studies of Silk Road." Digital Humanities 2016:Conference Abstracts.

Kitamoto, A., and Nishimura, Y. (2014). "Data criticism: general framework for the quantitative interpretation of non-textual sources." Digital Humanities 2014:Conference Abstracts.

Kitamoto, A. (2015). "MemoryHunt: a mobile app with an active viewfinder for crowdsourced annotation through the re-experience of the photographer", Fifth Annual Conference of the Japanese Association for Digital Humanities: Conference Abstracts.

Shepard, R.N., Metzler, J. (1971). "Mental rotation of three-dimensional objects." Science, 171(3972): 701-703.

Shift (2010). “Historypin.” http://www.history-pin.org/, (accessed on April 6, 2017).