Open vs. Closed: Changing the Culture of Peer Review

Despite our differing research methodologies,
subjects, and motives, the one thing scholars
across the disciplines and around the world
might agree upon is the significance of peer
review. Peer review may be the sine qua
non of academic work; we use it in almost
everything we do, including grant and fellowship
applications, hiring and promotion processes,
and, of course, in vetting scholarly work for
publication. We all operate under the agreement
that peer review is a good thing, by and large,
both a means of helping scholars improve their
work and a system for filtering that work for the
benefit of other scholars.
However, as I argue in
Planned Obsolescence
,
the means by which we conduct peer review
demand careful reconsideration and revision as
academic publishing moves increasingly online.
Clay Shirky has argued that the structures of the
internet demand a “publish, then filter” process,
encouraging the open communication of
all
of
the results of scholarly investigation, followed
by a process that filters those results for quality
(Shirky 2008). I explore the reasons such a
transformation is desirable at length in
Planned
Obsolescence
, primarily that it makes little
sense to replicate a mode of review designed
for print’s economics of scarcity within the
internet’s economics of abundance (see Jensen
2007); if what is scarce in the age of the network
is not the means of production but the time and
attention available for consumption, the best use
of peer review would be to help researchers find
the right text, of the right authority, at the right
time.
A born-digital system of review would work with
rather than against the strengths and values
of the network by privileging the open over
the closed, and by understanding the results
of peer review as a form of metadata enabling
scholars to find and engage with research in
their fields. How to build and implement such
a system, however, remains in question: how
do we devise a networked review system that is
open, honest, and thorough, that draws the best
from the “wisdom of the crowds” (Surowiecki
2004; Anderson 2006) while upholding the
standards that review is meant to serve?
Several examples of online review processes
already exist; within humanities journal
publishing, the most significant may be that
of
electronic book review
; articles submitted
there are posted in a password-protected review
space, where registered users of the site can read
them, leave glosses, and recommend acceptance
or rejection. However, though the editors use my
term “peer-to-peer review” in describing their
system, it falls a bit short of the truly open
system I imagine; the review system is still kept
behind the scenes, and while the reviews are
crowd-sourced, the reviewers producing them
aren’t asked to take responsibility for their
opinions by expressing and defending them in
public. This aspect of peer-to-peer review is key;
just as the quality of the algorithm determines
the quality of a computational filtering system,
the quality of the reviewers will determine the
quality of a human filtering system. Online peer
review must made open and public not just as
a means of increasing communication but as
a means of increasing reviewer accountability,
providing for the ongoing review not just of texts
but of reviewers.
In order to experiment with the possibilities for
an open review system, and with the consent of
NYU Press and my editor, Eric Zinner, I placed
the entirety of
Planned Obsolescence
online in
late September 2009. The text was published in
CommentPress, a WordPress plugin developed
by the Institute for the Future of the Book, which
enables the discussion of texts at a range of levels
of granularity, from the paragraph to the page
to the document as a whole. At the same time,
NYU Press sent the project out for traditional
peer review.
Such experiments have been conducted before;
in 2008, Noah Wardrip-Fruin published a
draft of
Expressive Processing
through
Grand
Text Auto
, while MIT Press sent it to outside
readers. Noah, however, wasn’t seeking to create
a head-to-head contest between closed and
open review; he was motivated by the desire
for feedback from a community he trusted
(see Wardrip-Fruin 2009b). My motives were

2
a bit more complex; I wanted that same
community-based feedback, but I also wanted
to test open review against more traditional
reviews, to gauge differences in the kind and
quality of responses produced within an online
system, and to project the kinds of changes to
CommentPress that might help transform the
plugin into a viable mechanism for peer-to-peer
review.
In slightly less than six months,
Planned
Obsolescence
received 205 comments from 39
different readers (not counting my own 78
responses). These comments are by and large
excellent, and have been extremely helpful in
thinking about the revision of the manuscript.
Most of the comments, however, are
locally oriented; CommentPress’s paragraph-
level commenting strategy encourages a close
attention to the particulars of a text rather than a
more holistic discussion. This focus on the text’s
details in the comments wasn’t unexpected; we
anticipated that the traditional reviews, being
written after the entire manuscript had been
read, would tend to focus a bit more on the
big picture than would comments made in
medias res. This assumption did largely bear
out; the offline reviewers tended more toward an
assessment of the overall argument.
Our first tentative conclusion, then, was
that a functional open review system would
require clearer ways for online reviewers
to leave broader comments. An update to
the CommentPress plugin, released a couple
of months into our experiment, helped
provide that functionality by highlighting the
“community blog” section of the site, which in
theory would allow members of a community of
readers to engage one another in discussion of
their reviews and of the project as a whole. In
actual practice, however, that engagement did
not occur, though it remains unclear whether
this is due to the blog’s belated introduction or
some other issue.
Additionally, however, Zinner asked the offline
reviewers whether they would be willing
to participate in our process, allowing us
to post their reviews for discussion and
response; one, Lisa Spiro, agreed. Spiro’s
willingness to participate, and the generosity
of her review, revealed the importance of
the social commitments involved in the peer
review process. Those scholars who have long
undertaken the often thankless work of peer
review have largely done so out of a commitment
to the advancement of knowledge in the field.
But fostering participation in online discussion
requires not just intellectual interest on the
part of individuals but also a solid, committed
social network. Reviewers participating in an
open process must have a stake in that process
beyond that of the disinterested reader; they
must understand the text and its author to be
part of a community in which they are invested
and to which they are accountable.
Beginning in March 2010, MediaCommons
will conduct another open review experiment,
publishing a small group of papers being
considered for a special issue of
Shakespeare
Quarterly
. Through this experiment, we hope
to explore a number of variables: the relative
weights of commitment to subject matter
and commitment to digital methodologies in
determining participation in open review, the
level of engagement in the review of article-
length (as opposed to book-length) texts online,
and the structures of participation in the review
of work by multiple authors in one venue.
Both experiments involve the review of
comparatively traditional forms of scholarship,
the book and the journal article, which we
have opted to begin with for two reasons: first,
that transforming the processes of reviewing
these forms of scholarship presents the broadest
potential impact on academic publishing as
it exists today, and second, that it confines
the question under consideration to
mode
of
review, rather than expanding into
criteria
for
review. That last is extremely important; many,
if not most, scholars working in new forms of
multimodal scholarship have encountered the
sense that the academy in general does not
know how to review such work. We hope to
experiment in the future with models for review
of new forms of scholarship.
This paper will, in the end, argue that a truly
effective peer-to-peer review system will need
to place its emphasis not just on developing
the technological network but on developing
the social network; it must be focused around
clusters of scholars who are already in dialogue
with one another. It must also be accompanied
by a shift in values that encourages scholars to
understand the business of reviewing as being
a commitment not just to the advancement