Check! - An online tool for the recognition and evaluation of DH work

Introduction

There has been an increase in the recognition of Digital Humanities (DH) projects as legitimate forms of Humanities research (MLA 2007). Although much work has been done to convince our colleagues that digital research is worthy, a pending problem is that they do not necessarily have the knowledge or tools to evaluate DH work (Rockwell, 2011). Most DH projects are not presented in traditional research output formats such as articles or books, but take on various formats such as development of metadata, textual markup, tools, websites and others (Schreibman et.al, 2011), so many evaluation committees are at loss with how to deal with these materials. At the same time, people developing a DH project do not necessarily know what elements should be included.
Although there are places in the world where advanced and sophisticated DH projects exist, the vast majority are individual and modest efforts with little access to or knowledge about best DH practices. This issue becomes particularly acute in emergent DH communities, such as Mexico, where there is little accumulated practical experience. In previously held workshops of the RedHD (Galina 2012; Galina & Priani 2011), where issues related to developing and promoting DH in the region were addressed, we concluded that one important aspect was providing both DH creators as well as evaluators with mechanisms for recognizing and evaluating the importance and impact of DH projects.
We therefore decided to develop an online tool for the evaluation of DH project in order to promote the recognition and evaluation of DH work, consisting of a set of guidelines coupled with an online checklist that allows users to evaluate a particular resource using an interface that displays the results of the evaluation, indicating areas of weakness and of strength. The aim of this tool is threefold: step-by-step evaluation tool for committees, a resource for developers and as an informal compliance of a minimum standard.
Methodology

A literature review of relevant guidelines was undertaken and an ad hoc committee was created to discuss the findings. Other tools were found to be either too specific or general (MLA 2012; Presner 2012; MLA 2011; Warwick 2007; Unsworth 2001) and additionally they were all provided as static document-based resources.
Based on this, guidelines were proposed and divided into sections: Authorship and Attribution, Documentation, Quality Control, Rights Management and Visibility and Dissemination, were created. We then extracted simple yes, no or not applicable questions in order to build up a checklist that corresponded to the guidelines. The checklist was then transferred online and we developed a punctuation system depending on the answers. The results are then displayed by section and overall performance.
The tool was reviewed by an expert group of 16 people. Each person was assigned five DH projects to evaluate from a random sample taken from the RedHD DH project database. A follow up meeting with all participants was used to discuss their experience and several modifications were made to the tool until a consensus was reached.
Conclusions

Creating an online resource for evaluating and building a DH project was a necessary but challenging endeavor. DH resources are heterogeneous and their objectives vary widely. It is therefore difficult to condense into a checklist DH desirable characteristics that are not too specific but at the same time manage to incorporate issues that are particular to DH projects and not to web-based projects in general. In addition we wanted our tool to address different types of audiences: creators and evaluators of DH resources whom would have different degrees of DH knowledge and expertise. Choosing simple and clear language for the best practices was a challenge. In addition, as we were able to ascertain from the discussions during the committee meetings, there are no clear and definite answers and this tool will probably have to be adjusted continuously as DH work and the technologies evolve and will require continuous community consensus. In the following months the tool will be tried out with two other user groups: a group of inexpert colleagues who are developing for a DH project for the first time and an evaluation committee that provides small funding grants for UNAM projects. The feedback from these exercises will also be incorporated as the work on this tool continues. We do believe however, that this tool that provides an informal certification of compliance will aid in the process of recognition of the validity of DH projects as it gives both creators and evaluators a general standard to which DH projects can be measured against.
References

Galina, I.Retos para la creación de recursos digitales en las Humanidades, El Profesional de la Información, 21(2), pp185-189, 2012 (ISSN: 1386-6710)
Galina, I., Priani, E., López, J., Rivera, E., Cruz, A.Tejiendo la Red HD- A case study of buidling a DH network in Mexico, Digital Humanities 2012, Conference abstracts, Hamburg, Germany. 16 - 22 July 2012, ISBN 978-3-937816-99-9, pp.456-458
MLA (2007). Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion, 2007, Modern Language Association, p.71
MLA (2011), Guidelines for Editors of Scholarly Editions, Modern Language Association, 2011.
MLA (2012), Guidelines for Evaluating Work in Digital Humanities and Digital Media, Created 2000, last reviewed 2012.
Presner, T. (2011). How to evaluate digital scholarship, http://humanitiesblast.com/Evaluating_digital_scholarship.pdf
Rockwell, G. (2011) On the Evaluation of Digital Media as Scholarship, Profession, 2011, pp. 152-168, 10.1632/prof.2011.2011.1.152
Schreibman, L., Mandell, L., et. al., (2011) Evaluating Digital Scholarship – Introduction, Profession 2011, pp. 123-201, 10.1632/prof.2011.2011.1.123
Warwick, C; Terras, M. et. al.(2007) Evaluating Digital Humanities Resources: The LAIRAH Project Checklist and the Internet Shakespeare Editions Project. In: (Proceedings) ELPUB 2007.
Unsworth, J. (2001) University of Virginia, Evaluating Digital Scholarship, Promotion & Tenure Cases. http://artsandsciences.virginia.edu/dean/facultyemployment/evaluating_digital_scholarship.html
UNAM, Disposiciones Generales para la Actividad Editorial de la UNAM, cited in López, C. and Estrada, A., Edición y Derechos de Autor en las Publicaciones de la UNAM, 2007.
UNAM, Recursos web – Lineamientos CATIC. http://recursosweb.unam.mx/recursos-web/lineamientos-unam/