Reforming Digital Historical Peer Review: Guidelines for Applying Digital Historiography to the Evaluative Process

Reforming Digital Historical Peer Review: Guidelines for Applying Digital Historiography to the Evaluative Process
Sternfeld, Joshua, National Endowment for the Humanities, United States of America, jsternfeld@neh.gov
From teaching students how to vet online websites, to formal peer review of digital publications, to evaluation of scholarship for tenure review, the need for a rigorous methodology to evaluate digital historical representations has never been more apparent. Sophisticated databases, digital libraries and archives, and virtual reconstructions have collectively reshaped engagement with the past that has challenged the boundaries of traditional historical practices. University professors encourage students to create mash-ups of historical multimedia content to be posted as YouTube documentaries. Museums employ mobile applications enriched with augmented reality to draw visitors further into an exhibition. Geospatial visualizations and virtual architectural reconstructions bring the past alive and challenge entrenched scholarly and popular perceptions.

While attention and resources have (justifiably) been focused on content creation and tool development, the digital history community has, until recently, neglected the development of a methodology that can evaluate digital work while meeting the needs of this shifting landscape. This has begun to change, as leading scholars in digital history have taken up the clarion call for reform of the peer review process. The perception is that peer review has become outdated and stagnant, promoting conservative scholarship while also failing to exploit more dynamic means of communication and commentary through social media and Web 2.0 technologies. Robert B. Townsend, in a American Historical Association (AHA) blog posting proclaims, “The challenge lies in developing new forms of peer review better fitted to the online environment, both before publication (in the development and assessment stage) and after publication (as a means of validating the value and quality of the work).” Robert B. Townsend. “Assessing the Future of Peer Review.” AHA Today. 7 June 2010. http://blog.historians.org/profession/1065/assessing-the-future-of-peer-review. The absence of peer review standards was also captured in a recent article for The Chronicle of Higher Education and online commentary. Jennifer Howard, "Hot Type: No Reviews of Digital Scholarship = No Respect," The Chronicle of Higher Education (2010), http://chronicle.com/article/Hot-Type-No-Reviews-of/65644/, accessed 31 October 2010. See also Dan Cohen. “Peer Review and the Most Influential Publications.” 19 October 2010. http://www.dancohen.org/. For a more comprehensive discussion of digital peer review, albeit slightly outdated in its conclusions, see Digital Scholarship in the Tenure, Promotion, and Review Process, ed. Deborah Lines Andersen. Armonk: M.E. Sharpe, 2004. While Townsend and others have highlighted the ills of the current peer review system, they have yet to propose a set of review guidelines or methodology that would bring together the interests and special knowledge of multiple disciplines.

In my DH2010 presentation, “Thinking Archivally: Selection, Search, and Reliability as a Framework for a New Digital Historiography,” I proposed a framework for evaluating digital historical representations called digital historiography.The abstract for the DH2010 presentation may be found online in the conference program. Digital Humanities 2010: Conference Abstracts. Eds. The Alliance of Digital Humanities Organisations, The Association for Literary and Linguistic Computing, The Association for Computers and the Humanities, and The Society for Digital Humanities. London, 7-10 July 2010. http://dh2010.cch.kcl.ac.uk/academic-programme/abstracts/papers/pdf/ab-747.pdf. Accessed 13 March 2011. An expanded version of the paper will appear in a forthcoming issue of American Archivist. Digital historiography is the interdisciplinary assessment of digital historical representations across diverse formats. It promotes the rigorous and coherent use, recombination, and presentation of historical information through digital technologies. Digital historiography also accounts for the increased reliance on complex information systems to organize and represent historical data. The merging of historiography with archival theory and new media studies – along with numerous other fields such as museum studies, information science, and other humanities disciplines – ensures a comprehensive examination of a representation.For a history and examination of the contemporary “divide” between the historical and archival professions, see Francis X. Blouin, Jr. and William Rosenberg. Processing the Past: Contesting Authorities in History and the Archives. Oxford University Press (2011).

Three fundamental areas of archival theory were raised in last year’s presentation as fundamental to unlocking the trustworthiness and soundness of a representation: content selection, search functionality, and metadata. Through a series of illustrative examples of historical representations it was shown that these areas convey a representation’s historical contextualization, where context is defined as the formal and humanistic relationships among data and resources. For an introduction to the relationship of archival theory and context, see for example, Jennifer Meehan, "Towards an Archival Concept of Evidence," Archivaria 61(2006), 143. The discussion of archival context has been subsumed in a larger, more general area referred to as “information as evidence.” For further discussion of this concept, see Terry Cook, "Archival Science and Postmodernism: New Formulations for Old Concepts," Archival Science 1(2001); Terry Cook, "What Is Past Is Prologue: A History of Archival Ideas since 1898, and the Future Paradigm Shift," Archivaria 43(1997); Margaret Hedstrom, "Archives, Memory, and Interfaces with the Past," Archival Science 2, no. 1-2 (2002); Joanna Sassoon, "Beyond Chip Monks and Paper Tigers: Towards a New Culture of Archival Format Specialists," Archival Science 7, no. 2 (2007); Jennifer Meehan, "Making the Leap from Parts to Whole: Evidence and Inference in Archival Arrangement and Description," American Archivist 72, no. 1 (2009).

Whereas last year’s paper focused on the theoretical underpinning of digital historiography, this year’s presentation will provide a programmatic framework that scholars, archivists, librarians, curators, editors and technical specialists may adapt and apply to their own work. There are already a number of promising developments underway in establishing a set of guidelines and practices for digital humanities evaluation, particularly in literature; nonetheless, a similar approach in digital history has been lacking.Perhaps the most comprehensive guide to evaluating digital scholarship can be found with the MLA Guidelines for Editors of Scholarly Editions. Section V is devoted to electronic scholarly editions, and prompts reviewers to consider elements of a digital work such as TEI encoding and the user interface. An NEH-funded Summer Institute entitled “Evaluating Digital Scholarship” will be hosted by NINES at the University of Virginia 30 May – 3 June 2011, which will expand upon a one-day workshop held at Digital Humanities 2010 in London.

This paper will provide a guide to evaluating digital historical representations using a series of exploratory questions. The following questions may provide an entry point with which to kick-start an analysis. It should be noted that these questions do not evoke clear-cut responses, but rather require deeper interpretation of the representation:

Is the representation’s historical content comprehensive or representative of the period/event/issue in question?
How do metadata schema and other descriptive information shape the interpretative possibilities of the representation?
What capacity does the user have to repurpose historical data for additional study? To what extent does a user have sufficient contextual information such as the content’s provenance to conduct such repurposing?
How does the user search or navigate within the representation? In what ways does the interface facilitate or prohibit advanced humanistic inquiry?
These questions apply a combination of historical and archival understanding to address a representation’s approach to selection, search, and metadata. At a more general level, they suggest the need within digital historiography to develop a peer review methodology that examines the intersection of technological and humanistic components of a representation.
A Three-Axis Framework for Peer Review
This paper will propose, based on the sample questions above, three interlocking axes for a peer review methodology: Historical Content, User Experience, and Creator Intent. Critiques of digital representations have all too often isolated one axis at the expense of considering the others. Scientifically driven user or human-computer interaction studies may overlook the unquantifiable nuances of historical contextualization. Similarly, an analysis of content often neglects a representation’s user interface and how it may affect information access. Perhaps the most disregarded of the three areas is the creator’s intent for constructing a digital historical representation. In the fierce competition for funding in the digital humanities, developers must be able to justify their resource-intensive project, which raises basic questions surrounding the representation’s purpose and its contribution to historical scholarship or programming.

For the purpose of demonstrating how to apply digital historiography to peer review, a single digital historical representation will be selected and the audience will be guided through a brief, yet systematic evaluation. In terms of historical content, we will consider the representation’s engagement with relevant scholarship. Although a digital representation’s format may seem to belie traditional modes of historiography, it will be argued that a representation’s demonstrated recognition of historiographic trends is essential for establishing whether it advances new areas of historical understanding. Unlike a text-based monograph, analysis of content depends on the capacity of the user to generate, not just consume, plausible knowledge; therefore, peer review may focus more on the questions that may be posed through a representation rather than the construction of a single argument.

In terms of user experience, this presentation will focus on the question of access to historical content. More than just a scientific optimization of keywords, a reviewer must consider the user experience as a culmination of design features, tool functionality, and search capability. This raises issues regarding the application of standards and best practices. The selection of an appropriate metadata schema or digitization standard will determine a representation’s interoperability with other resources, which will in turn influence historiographic assessment.

Finally, consideration of authorial intent, embedded within elements such as a site’s introductory statement or contextual essays, provides a third axis that can anchor both a representation’s content and user experience. The core question to be considered here will be the representation’s intended or anticipated audience. Digital technology has enhanced access to historical knowledge for a wider public, which has placed additional responsibility on the part of the creator to consider how historical information is selected, presented, and most importantly, handled by diverse types of users.

The Benefits of Peer Review to Digital History
A peer review system that follows shared and accepted standards and methodology may have significant benefits for advancing digital historical scholarship and digital humanities infrastructure in general.For further discussion of how the digital humanities must re-consider infrastructural needs, see Christine L. Borgman. “The Digital Future is Now: A Call to Action for the Humanities.” Digital Humanities Quarterly. Fall 2009. Volume 3, Number 4 Representations may be assessed on the merits of content and usability. In terms of scholarship, digital historiography may liberate scholars and developers to consider how to harness technologies in service of historical inquiry when all too often the reverse seems to hold true. The time for experimentation for its own sake has passed, and we should begin to consider how technology might contribute to more sustainable development of innovative modes of scholarship.

This presentation is not intended to provide a fixed method for peer review; rather it will encourage a revival of humanities-driven interpretative analysis that addresses central areas of scholarship, audience, argument, and most importantly, inquiry. Peer review must be contingent on the unique set of questions related to a representation’s subject area and formal qualities. Those in attendance will have the opportunity to take away a set of general questions that they may use to devise their own review criteria, or that may stimulate further dialogue about the peer review process. Perhaps most significantly, this presentation will defend the need for collaborative, transparent review that brings together subject and technical specialists, which can spark evaluation of noteworthy digital work that continues to elude mainstream academic recognition.The thoughts and ideas expressed in this paper are entirely my own and do not reflect those of the National Endowment for the Humanities or any other federal agency.

References: