Evaluating Digital Scholarship: A Case Study in the Field of Literature

Evaluating Digital Scholarship: A Case Study in the Field of Literature
Schreibman, Susan, susan.schreibman@gmail.com Digital Humanities Observatory,
Mandell, Laura, laura.mandell@gmail.com
Olsen, Stephen, SOlsen@mla.org
Evaluating and receiving credit for digital scholarship within traditional disciplinary areas in the humanities has been a concern much discussed, not only within the digital humanities community, but at think tanks on the future of scholarly publishing, within institutions, at professional associations in the various disciplines of the humanities, and in journal and newspaper articles.For just a small sampling of recent postings and articles see Scott Jaschik, ‘Tenure in a Digital Era’. Inside Higher Education (26 May 2010); ‘Evaluating Digital Scholarship, Promotion & Tenure Cases’, Office of the Dean, University of Virginia, (link) ; ‘Guidelines for Evaluating Work with Digital Media in the Modern Languages’, Modern Language Association, (link) ; Julia Flanders, ‘The Productive Unease of 21st-century Digital Scholarship’. Digital Humanities Quarterly (Summer 2009); Jeanne Glaubitz Cross, ‘Reviewing Digital Scholarship: The Need for Discipline-Based Peer Review’. Journal of Web Librarianship (December 2008); Joan F. Cheverie, et al, ‘Digital Scholarship in the University Tenure and Promotion Process: A Report on the Sixth Scholarly Communication Symposium at Georgetown University Library’ Journal of Scholarly Publishing (April 2009) 219-230.

Over the past few years The Modern Language Association has taken the lead in encouraging the recognition of digital scholarship in promotion and tenure cases. Its 2006 Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion ( (link) ) offered unequivocal support of digital scholarship. Among its recommendations are the following:

The profession as a whole should develop a more capacious conception of scholarship by rethinking the dominance of the monograph, promoting the scholarly essay, establishing multiple pathways to tenure, and using scholarly portfolios . . . .[and] departments and institutions should recognize the legitimacy of scholarship produced in new media, whether by individuals or in collaboration, and create procedures for evaluating these forms of scholarship.Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion. (link)
This report and others (see "Our Cultural Commonwealth: The report of the American Council of Learned Societies Commission on Cyberinfrastructure for the Humanities and Social Sciences" etc.) make it clear that the Humanities must broaden traditional definitions of scholarship and reconceptualize its methods of evaluation. But it still falls to us in the digital humanities to articulate the scholarly content and value of our work and to propose explicit procedures for effectively evaluating it.

In 2008 the MLA’s Committee on Information Technology released an Evaluation WikiThe Evaluation of Digital Work. (link) (based on the work of Geoffrey Rockwell who served as a member of the Committee from 2005-08) that provides a framework for departments to evaluate this new scholarship. As a result of this work, the authors of this paper led a workshop in evaluating digital work for tenure and promotion at the 2009 MLA convention in Philadelphia. The workshop was designed to provide both a framework from within which digital scholarship could be evaluated, as well as a forum for the authors to evaluate just how difficult it was for non-specialists to come to terms this new work. With this in mind, several themes were addressed in the workshop design:

The impediments involved in evaluating digital scholarship, particularly in terms of interdisciplinarity, authority, and use of standards;
Issues of disciplinarity: digital scholarship does not fit comfortably into the what is currently valued in the profession;
The absence of expertise at the departmental level in evaluating digital scholarship handicaps both the evaluators and the candidates;
The intellectual stakes of such work;
The value and credit accorded to interdisciplinary, multi-institutional, collaborative work.
In the field of literary studies the gold standard for tenure and promotion is the publication of the monograph. Editorial scholarship -- textual criticism, book history, scholarly editing, the sociology and bibliography of texts --particularly in North America, has been consistently devalued since the rise of literary theory. That the digital is conducive to the kinds of projects that have been denigrated by the academy (certainly since new literary criticism), including pedagogy, public humanities, and the creation of scholarly editions, has made the argument for including this work in tenure and promotion cases all the more difficult. These aforementioned themes and questions were meant to engage the group in questioning such values.

The workshop was structured around case studies based on the experiences of individuals in the field. In some instances the case studies were anonymized (as several of the workshop leaders created studies based on their own scholarship), others were not (as when it was clear who the author was from the online scholarship under discussion).

Attendees were seated at tables of eight and for each of three rounds had two case studies to choose from. The first round of case studies were what might term fairly traditional digital scholarly editions. This type of digital scholarship was chosen because it most resembled analogue scholarship. Nevertheless, when we opened the conversation up for discussion, the opening remark by a Departmental Chair questioned that the edition under discussion should be counted towards scholarship at all, as in her opinion, the creation of scholarly editions fit squarely into the category of service. This became a recurring theme of the individuals who did not understand the theoretical and technical imperatives behind the scholarship consistently categorized the work as service. It became clear that the production of editions, whether distributed in analogue or digital form, were not viewed as scholarship on par with critical theory.

Moreover, when the authors of case studies explained the theoretical methodology implicit in their research in terms of the technologies used, non-technical participants consistently complained that the authors were engaging in techno-speak jargon. Participants also complained that the work was undertheorized: however, when it was pointed out that theories common to digital scholarship (for example, the limitations of particular metadata schemes, issues arising from the hierarchical nature of XML, or the impediments to using a standard such as TEI for genetic editing) were evoked and challenged, these participants felt that it was not sufficient to rely on theoretical perspectives not normative to the discipline of literature (i.e. coding was not satisfactory as a basis for the field).

Subsequent case studies teased out issues of the implications of, in effect, outsourcing the evaluation of scholarship to (scholarly) presses and journals, the vast majority of which have not developed an economic model for publishing digital scholarship. Discussions circled around alternative models of validating digital scholarship, focusing on the work, for example, of NINES. Indeed, the success of the NINES model was so apparent in these discussions that NINES has committed its next two Summer Institutes (generously funded by the National Endowment for the Humanities) to evaluating digital scholarship.

Although the writing is on the wall that our field must change to encompass new scholarship and new scholarly forms as engendered by the new technologies, the impediments to evaluating this scholarship by non-specialists are paramount. The digital archive may have been the first form to challenge the primacy of print scholarship, but new forms of scholarship are emerging that will even more radically challenge the status quo.

It is clear from the research carried out by the authors in the preparation and delivery of this workshop, the creation of the CIT Guidelines, and the development of the NINES Summer Institutes, that it is imperative for the Digital Humanities community to take a more proactive role in supporting departments as they are increasingly called upon to evaluate digital scholarship. This burden cannot continue to be borne solely by each individual candidate, as is currently the norm. The overriding question for these activities must be how do we provide a framework so that evaluators can evaluate the research within the technological context within which the scholarship is undertaken.

The discussion of the case studies by the participating department chairs, senior faculty members, and junior faculty members revealed to us some of the specific difficulties departments and institutions have in recognizing scholarly activity in some of its new digital forms:

The evolving definitions of scholarship in language and literature over the past 50 years (Eagelton Terry Eagelton. "The Functions of Criticism," How To Read a Poem (Malden, MA: Basil Blackwell 2007), 1-24.), in particular the conflict between criticism and philology as the dominant mode of scholarship;
The discounting of scholarly activities like textual editing and translation, and the mislabeling of much digital scholarship as serviceConnie Moon Sehat and Erika Farr. ‘The Future of Digital Scholarship: Preparation, Training, Curricula Report of a colloquium on education in digital scholarship’. (2009). (link) (McGann,McGann, Jerome. “A Note on the Current State of Humanities Scholarship.” Critical Inquiry 30/2. (link) GablerGabler, Hans Walter. "Theorizing the Digital Scholarly Edition." Literature Compass 7/2 (2010) 43-56.);
The ongoing crisis in scholarly book publishing (Waters,Lidnsay Waters. “A Modest Proposal for Preventing the Books of the Members of the MLA from Being a Burden to Their Authors, Publishers, or Audiences.” PMLA 115 (2000): 315–17. Guillory,Guillory, John. "Evaluating Scholarship in the Humanities: Principles and Procedures." ADE Bulletin 137 (Spring 2005), pp. 18–33. GreenblattGreenblatt, Stephen. “A Special Letter.” 28 May 2002. 1 Nov. 2010 (link) )
Our paper will summarize what happened at the workshop, in response to which we will present action items to be undertaken by organizations such as MLA and ACH.