Arduino Circuits and Javanese Puppets: 'Re-materializing' Digital Archives through Tangible Interfaces

The creation and analysis of digital collections is a key aspect of the digital humanities. Deciding what to archive, how to organize the documents, and how to present them to the public are never easy questions. But in the case of theatre and performance archives, these questions are especially complex since theatre performances are necessarily transient and they depend on the embodied co-presence of actors and spectators. In order to document the embodied and ephemeral nature of theatre performances, digital archives often try to include a wide array of documents (video recordings, photographs, motion capture data, 3d models of theatrical spaces and technical scripts). The makers of digital theatre archives are at pains to contextualize their materials and account for aspects that cannot be easily documented and transmitted. Can interaction design help communicate these aspects? Is it possible to imagine and construct interfaces that can communicate cultural context, transcending the limitations of a computer screen, mouse and keyboard?
To investigate these questions within the context of an Indonesian performance archive, I developed a Tangible User Interface (TUI) using open source hardware (Arduino microcontrollers and an array of sensors) and culturally-coded objects that are important for the performances in the archive. This interface was developed as an interactive artwork for educational museums and as a thought experiment on how tangible, culturally-specific interfaces can constitute instances of academic research outputs in the digital humanities.
The archive in question is the Contemporary Wayang Archive (CWA,
http://cwa-web.org), a collection of digital recordings and metadata pertaining to Javanese wayang kulit (shadow puppetry), which I have been developing with my collaborators at the National University of Singapore since 2012. Wayang kulit is the oldest theatre form of Indonesia and one of the most important theatre traditions of Southeast Asia. It consists of a form of puppetry where a dalang (narrator-puppeteer) singlehandedly moves all the puppets, speaks all the character parts, jokes with the audience and directs the musicians. He is at the same time a puppeteer, a storyteller, an orchestra conductor and a stand-up comedian. In order to communicate with the audience and the orchestra, he uses different objects in order to control the progression of the story. For this project, I wired different sensors into these objects in order to develop a TUI that recreates the spatial setup of a conventional wayang performance and three key material components of this setup:

A kerlir or screen where videos from the CWA are projected.
The kayon. This puppet is shaped like a leaf and it has different narrative functions. In between scenes, the dalang rotates it around its axis and places it on a banana log at a specific angle (either 45⁰, 90⁰ or 135⁰) to indicate the progression of the story. This visual cue is important for audience members that don’t watch the entire show. A conventional performance lasts eight hours and the audience members often come and go, drifting in and out of attention. Depending on the specific angle of the kayon, knowing audience members can estimate the specific moment in the development of the performance (which is divided into three main acts). For my interface, I used an Inertial Measurement Unit (IMU) and a wireless sensor in order to detect rotation (angular velocity) in the puppet. By rotating an actual puppet, users of this interactive artwork can navigate to a different digital video in the collection.
The cempala. By hitting this wooden mallet against the puppet chest, the dalang cues the musicians to start and stop the musical accompaniment. For the interactive artwork, I wired the cempala to a Piezoelectric sensor to measure vibration. When the users of the artwork knock the cempala against a box that mimics the puppet chest, they can start and stop the videos and additional contextual information appears on the projection screen.
The usage of these objects is not exactly the same in this interactive artwork display as it is in an actual wayang performance. But it can invite users to think about the importance of materiality and embodiment for this particular theatre tradition. In such a way, this interactive artwork can be considered a piece of digital scholarship, a research output of the DH research into wayang kulit that complements the forthcoming online version of the archive and other publications that might arise from this research project (see, for example www.wayangkontemporer.com).
Although this artwork has a very specific origin and function, I hope it will resonate with scholars working in other areas of the digital humanities who are engaged in building and theorizing new artefacts for the communication of academic research. The objective of this short paper is to frame the creation of this artwork within the larger context of the digital humanities, exemplifying modes of scholarship that can emerge at the intersection of cultural knowledge, open hardware and digital technologies.

Figure 1. Conventional wayang kulit setup

Figure 2. IMU sensor attached to the kayon (left) and piezoelectric sensor attached to the cempala (right)

Figure 3. A user interacting with the TUI

Bibliography

Fishkin, K. P. (2004). A taxonomy for and analysis of tangible interfaces.
Personal Ubiquitous Computing,
8(5): 347–58.

Bonanni, L. et al. (2010). Tangible interfaces for art restoration.
International Journal of Creative Interfaces and Computer Graphics,
1(1): 54-66.

Jones, S. et al. (2009). Redefining the Performing Arts Archive.
Archival Science,
9(3): 165-71.