XR in DH: Extended Reality in the Digital Humanities

Driven by the gaming and entertainment markets, Extended Reality (XR), which includes Augmented Reality (AR) and Virtual Reality (VR), continues to expand rapidly in both its capabilities and its consumer adoption. Once the sole province of large installations, rack computers and bulky and expensive headsets, a wide range XR / Augmented / Virtual Reality experiences are now available on a variety of platforms and appliances, whether as standalone, wireless headset units, or mobile apps.

A common thread of the use of XR representations and techniques is that they are at once both extremely complex and stunningly intuitive both to render and to interpret. The same paradoxicality can be said of some aspects of digital humanities research. Indeed, the questions that Digital Humanities research and pedagogy address are often highly complex, attributable sometimes as much to their technical difficulty as to the interdisciplinarity of thought they represent. As scholars seek to understand and embrace such complexities, they have begun to use tools that favor experiential, rather than narrative, representations of scholarly work, such as XR.
The ability for DH to flourish while comprising such internal contradictions suggests the capabilities of multidimensional technology to distill and refine the essential points of complexity by articulating them in those dimensions. In this manner, multidimensional scholarship seeks to reveal the underlying essence of DH projects by employing rich, deep and immersive experiences in pedagogy, data visualization, modeling and simulation.This panel will present recent research and pedagogy in literature, anthropology, and other humanities subfields using XR.

This panel explores the application of virtual and augmented reality technology in digital humanities research and education by a number of DH scholars. Dr.
Rachel Hendery
will describe work in spatial representations of text and linguistic data in VR, Dr.
Lynn Ramey
will present on the experience of a number of student groups using Unity and VR. Reflecting on several creative VR projects, Dr.
Mona Kasra
will explore the process of creating new collaborative, methodological, and pedagogical frameworks that facilitate hybrid research/practice projects in arts and humanities scholarship. Dr.
Amanda Licastro
will demonstrate the potential of using VR to teach literary analysis in the undergraduate classroom. Dr.
Geoffrey Rockwell
will look at the design of AR serious games and the learning both in the design and playing and Dr.
Victoria Szabo
will describe work at establishing evaluative mechanisms and criteria for XR projects.

“VR for outreach / VR for research”
Dr. Rachel Hendery
In a recently funded project about pre-colonial contact between Australia and the surrounding region, we have begun adapting a virtual reality project we developed previously for public engagement use, to recreate it as a platform that researchers can use to interrogate their data. The earlier version of the project, named
Glossopticon had been very successful in museums, university showcases, and other public events, for allowing students or the general public to explore Pacific islands, see a visual representation of the incredible multilingualism of the region, and to hear spatialised audio of the languages in question. The data was drawn from the PARADISEC database, and links to items in the database were automatically saved as the user explored the VR, being presented to them afterwards in a webpage, so in effect
Glossopticon functioned as a gamified user-interface to the database.

For the new project, which we have named
Layered Horizons, we wanted a way to be able to layer data relating to the linguistics, archaeology and anthropology of the region onto a 3D map, and for users to be alerted to relationships between locations that the data suggested might be important, in particular when these suggestions are the result of correlations between the different kinds of data. Through an iterative process involving collaboration between linguists, anthropologists, archaeologists, and designers, as well as successive user-testing phases, we have settled on a more abstract look and feel to the map, with point-cloud landscapes that can be warped and shifted to indicate relationships between place, and controls entirely based on natural gestures, interpreted by the LEAP Motion, rather than relying on controllers of any kind (c.f. Pescarin et al 2013 for some of the challenges and benefits of this latter approach.). In this talk I will explain these design decisions, showcase the latest version of the project, and discuss the ways in which the platform has enabled research discoveries so far.

Glossopticon

Layered Horizons

“Audience as Performer in Immersive Virtual Reality Experiences”
Dr. Mona Kasra
Immersive media such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) are shifting the long and complex relationship between audience and performer, stage and screen, and live and mediated performance. These emerging platforms continue to offer new experiential affordances that bring together performers and audience members in a 360-degree, virtual environment, reimagining the perceptual continuity of live performance.
In theater, dance, and other live performing arts, the liveness of the interaction between human performer and human audience is deemed essential as the audience stays in close physical and temporal proximity to the performer at the moment of their performance. In a mediated performance, however, the live interaction between the audience and the virtual performer is achieved only when the audience engages with disembodied digital data in real-time according to the size and configuration of their viewing devices. As Philip Auslander maintains, “the idea of what counts culturally as live experience changes over time in relation to technological change” (2012, 3).
In VR/AR/XR, liveness is realized through digital interaction and participation in the actual performance. By inviting audience members to partake, create, and become performers themselves, immersive experiences afford new experiential control and new sets of possibilities for live engagement. Surrounded by a multisensory and immersive environment, the audience can experience a mediated performance from different perspectives, reposition themselves within the space where the performance takes place, or even contribute to the performance itself. They can become performers, freely moving and interacting with the performance in real-time, using motion capture and sensing systems.
This paper explores the ways by which VR, AR, XR redefine the performer-audience relationship by pushing the boundaries of 'live' performance and extending immediacy and intimacy into the virtual space. Drawing on several creative VR project examples varying from immersive dance performances to interactive VR installations, I will examine the process of creating new collaborative, methodological, and pedagogical frameworks that facilitate hybrid research/practice projects in arts and humanities scholarship. I will describe some of the critical outcomes of these projects as they pertain to the ongoing debates concerning ‘presence’, ‘liveness’ and ‘virtual embodiment' and reflect on representational, affective, and creative possibilities of immersive media for arts and humanities research and teaching. This talk will also focus on the interdisciplinary nature of the medium and on how multimodal approaches offer new opportunities for practice-led research.

“Teaching Narrative and Literary Analysis with VR”
Dr. Amanda Licastro
In this presentation the speaker will explicate a series of assignments implemented in an undergraduate course which used Virtual Reality (VR) technology to teach literature. The aim of the experiment was to use VR in order to guide students through the process of understanding narrative structure and critical themes present in Mary Shelley’s
Frankenstein. In honor of the 200th anniversary of the 1818 version of the novel, students created VR applications inspired by one of the themes explored in the original work. Throughout the semester students read literary works and criticism such as Jennifer Haley’s The Nether, Donna Haraway’s The Cyborg Manifesto, and Lennard Davis “On Normalcy” along with viewings of The Stepford Wives and the “San Junipero” episode of Black Mirror [...] as well as “Lincoln in the Bardo,” the first VR application adapted from a novel
. These experiences culminated in a final project that instructed students to pitch their idea for an educational VR application that could be used to teach a theme from the novel. The proposal had to be clearly related to
Frankenstein, with accessible learning objectives backed by literary criticism. Students then worked in groups in create prototypes and a formal proposal for the top three projects. Each group presented their proposal to a local educational VR company, who then choose a winning project to turn into a full fledged VR application. This real world, client-based assignment prepared students from a wide range of academic disciplines to compose in a variety of multimodal genres while engaging in close reading and literary analysis. This presentation will feature the scaffolded assignments backed by pedagogical theory stemming from writing studies and DH scholarship. Sample student projects will be displayed, and selections from student reflection papers will be shared as evidence of the success of this project.

“VR, Unity, and Student Groups”
Dr. Lynn Ramey
Working with students on research projects involving virtual reality in the humanities classroom has become significantly easier in recent years. Free or inexpensive platforms and countless tutorials aid students with their assignments. Visual scripters allow students to make a VR world with little or no programming experience. VR is rapidly becoming more portable, delivering the experience to any classroom without excessive or expensive technology via emerging platforms and gaming devices readily available in many households.
Despite the incontrovertibly rosy outlook for VR in educational environments, obstacles still exist. As projects become more and more complex, more powerful computers with better graphics cards and extended storage are needed to make a smooth experience for users—technology that is not readily available to all students at every institution. While visual scripters can certainly get a reasonable project started, programming skills are imperative for accomplishing the complex goals of certain student projects. Even when the environment has been perfected on a student computer, the steps to build the project for a particular device are complicated and can require specific licensing. Finally, students can find teamwork and communication challenging, and putting together an appropriate team for a VR project involves students with very different skills and interests working together, often outside of class while juggling busy schedules.
The communication will conclude with future directions for use of VR in the humanities classroom. What lessons were learned from these classes, and how can we provide a meaningful and reflective experience for humanities students of various backgrounds? What tools were most useful, and what remains on the wish list for educational VR development?

“Campus Mysteries: Playing with Serious Augmented Reality Games”
Dr. Geoffrey Rockwell
With the release of Pokémon Go (Niantic 2016) Augmented Reality (AR) games have gone mainstream. AR games on mobile devices use the built in GPS to move play out into the community letting players “catch” items at locations in a treasure hunt style of play. The Campus Mysteries project at the University of Alberta was an experiment in designing AR serious games for learning local history that started in 2009.
In July of 1918 the Spanish Flu hit Canada. This epidemic was one of the worst in human history killing between 20 and 40 million people around the world. It came to Edmonton, Alberta in October of 1918 and the University of Alberta closed down for around two months. The Campus Mysteries project created a ghost hunting game for summer students to teach this local history. The game was developed by a team of Digital Humanities, Education and Computing Science faculty and graduate students. It was developed with an AR authoring environment called fAR-Play (for Augmented Reality Play) which we iteratively developed along with the games.
In this short presentation I will talk about how we evaluated Campus Mysteries game and other AR games developed. The AR version of Campus Mysteries was tested against two control versions, one that used QR codes and one that used paper. The experiment showed the viability of AR serious games for learning, but also showed how the technology can be a hurdle if it isn’t used properly. A further limitation is the treasure hunt paradigm which most locative games use. We need an expanded repertoire of understood game genres to better match the issues dealt with. On that note, I will conclude by talking about the development of the game and the development of the authoring environment as themselves a form of playful learning.

“Evaluating XR: Standards for an Emerging DH Medium”
Dr. Victoria Szabo
Evaluating scholarly work in XR poses a challenge to conventional academic practice in ways that are both familiar to DH practitioners and which amplify some of the biggest challenges to its wider acceptance as scholarly work. This talk outlines some of the Evaluation Best Practices initially proposed in 2013-14 and elaborated most recently in a 2018-19 Institute. Our goal is to augment the general guidelines provided by scholarly organizations like the MLA, CAA, and AHA around digital scholarship to account for the specific challenges XR offers. The contributors to the XR Evaluation Standards project come from a wide range of humanities disciplines, including Art History, Archeology, History, Journalism, Languages, Literature, Media Arts, and Music as well as Scientific Visualization, Computer Science, and Architectural Engineering. Their application interests include historical reconstruction, cultural heritage, digital storytelling, language instruction, representation of fictive spaces, media performance, and data visualization. They work in immersive CAVE spaces, high- and low-end headsets, Cardboards and mobile devices. Content includes 3D models of buildings, objects and landscapes, 360 video, abstract data visualizations, ad hybrid media installations.
The group converged around three key meta-principles for evaluation: Integrity, Integration, and Impact. Integrity relates to the ability of the material to interpret and transmit source materials, and the extent to which it surfaces an original argument. Interaction focuses on the extent to which the interactive elements aid in interpretation and discovery. Impact refers to the work’s originality, the extent to which it affects the conversation in the disciplinary field, as well as for the XR community. Added to these transdisciplinary principles for evaluation are considerations derived from the specific disciplines related to the medium of XR itself: art/design practice, info science, sci viz, communications studies, computer science, cognitive science. To what extent can and should practitioners using such tools be judged according to these specialist standards? Our team suggests a range of potential measures arising out of a more scientific perspective on the use of the technology itself. These include: user interface and experience design, graphic design, response time, accuracy, presence or absence of negative effects (motion sickness), biometric measures of “presence” in virtual spaces, memorability, comparative analysis with other forms, emotional impact and more. In terms of XR as a humanities endeavor, evaluation standards might include: the extent to which the project pushes against the usual constraints of the medium to signal ambiguity and conflicts, uncertainty, temporal change, the position of the observer, access to underlying data, and visible engagement with a scholarly argument. This dialog between scientific and humanistic communities becomes especially tangible within the collaborative contexts where XR work is being done. Our hope is that by surfacing a range of evaluation measures that draw upon the values of both communities, that we can further a conversation around what XR can do for us in the humanities, and what we in the humanities can do for it as an emerging media form.
References
Hendery:

Pescarin, Sophia et al. 2013. NICH: a preliminary theoretical study on Natural Interaction applied to Cultural Heritage contexts.
2013 Digital Heritage International Congress (DigitalHeritage) Vol 2: 28 October - 2 November
.

https://ieeexplore.ieee.org/document/6743760

Thieberger, N., & Barwick, L. (2012). Keeping records of language diversity in Melanesia, the Pacific and Regional Archive for Digital Sources in Endangered Cultures (PARADISEC).
Melanesian languages on the edge of Asia: Challenges for the 21st Century,
239-53.

Thieberger, Nick & Rachel Hendery. 2017. Sonorising and visualizing archive records.
5th International Conference on Language Documentation and Conservation (ICLDC), Hawaii.
Full paper available at http://hdl.handle.net/10125/41966

Kasra:

Arora, Gabo. (2018). ZIKR: A Sufi Revival. VR film,
Sundance Film Festival
.

Auslander, Philip. (2012). Digital Liveness: A Historic-Philosophical Perspective.
PAJ: A Journal of Performance and Art
, 34(3):3-1.

Iñárritu, Alejandro G. (2017). Carne y Arena. VR installation,
Cannes Film Festival
.

Licastro:
Davis, Lennard. Constructing Normalcy. http://blogs.fad.unam.mx/asignatura/adriana_raggi/wp-content/uploads/2014/05/Davis.pdf.
Haraway, Donna. A Cyborg Manifesto. http://www.egs.edu/faculty/donna-haraway/articles/donna-haraway-a-cyborg-manifesto/. Accessed 20 Aug. 2014.
Haley, Jennifer. Nether, The. Samuel French, Inc., 2015.
“San Junipero.” Black Mirror. Written by Charlie Booker, directed by Owen Harris, Netflix, 2016.
Shelley, Mary. Frankenstein, Second Edition. Edited by D. L. Macdonald and Kathleen Scherf, 2 edition, Broadview Press, 1999.
NYTVR application “Lincoln in the Bardo”
Rockwell:

Humphries, M. O. (2005). “The Horror at Home: The Canadian Military and the “Great” Influenza Pandemic of 1918.”
Journal of the Canadian Historical Association / Revue de la Société historique du Canada
, 16:1, p. 235-260.

Rockwell, G. and S. Gouglas (2019). “Experiments in Alternative / Augmented Reality Game Design: Platforms and Collaborations.”
Seeing the Past with Computers: Experiments with Augmented Reality and Computer Vision for History
. Ed. K. Kee and T. J. Compeau. University of Michigan Press, p. 158-175.