Action and Interaction in Music and New Media Art: Exploration of Musicians' Performative and Interactive Decisions as Evidenced by Annotated Musical Scores

Introduction
Preservation and representation of digital art presents a
significant challenge for curators, archivists and artists.
The most notable problem is the fundamentally interactive
nature of this evolving art form. Because there are no clearly
defined frameworks for the authentic representation of the
variations resulting from interactions, curators and archivists
are currently deciding on a case by case basis which interactions
should be preserved, who should be able to interact with a given
piece, and how to represent those interactions in a meaningful
and objective way. Not only is this method problematic from
a representational standpoint, it's very time-consuming. My
thesis will attempt to resolve some of these complexities by
making connections between this new art form and an
established one, namely music; assessing the representational
characteristics of music's notation system and exploring the
methods by which musicians and conductors handle musical
interaction.
Background
One of the most significant challenges for the new media
art community is development of a representation
framework for preservation purposes. There are currently three
preservation models under consideration. The first two have
technical origins, and should be familiar to the general digital
preservation community: migration, the premeditated upgrade
of file formats; and emulation, which focuses on development
of ur-operating systems able to run obsolete media. The third
option, much more radical, and developed by and for the new
media art community, is re-interpretation (Depocas, Ippolito
& Jones); a method intimately related to the presentation, exhibition, and performance of an interactive new media art
object.
While re-interpretation is essential to success in the performing
arts, in the fine arts and electronic preservation communities,
the idea of re-interpretation as a valid preservation strategy
challenges beliefs that are at the heart of these fields.
Specifically, for the purpose of archival preservation, the
characterization of a 'reliable' or 'authentic' object will need to
undergo a significant transformation if these new media art
objects; highly variable, interactive, and data-rich cultural
artifacts, stand a chance of meaningful survival. Although
radical, curators, archivists and conservators could regard the
idea of re-interpretation as a way to honestly address the
realities of presenting a variable work over time. Both migration
and emulation effect subtle re-interpretations of a work, whether
we perceive those changes or not. For example, the processor
speed of an emulated computer will display an interactive work
from the mid-1970s differently than the original operating
system would. Whether that difference fundamentally changes
the work's meaning can only be determined if a representational
framework exists in which the creator/cataloger has the ability
to address the realities of this medium's artistic creation and
representation process: specifically, the highly interactive nature
of the work, the variable quality of output, and technically
complex interactions within the work itself must be addressed.
By embracing the paradigm that includes re-interpretation as
a valid preservation strategy, digital preservationists will need
a thorough and consistent representation of the original from
which to work.
In mid-2004 Richard Rinehart delivered a paper arguing that
new media art is more like a musical performance than it is like
a painting or a book, and therefore more appropriately
represented by a scoring system than the text-centric methods
used today (Rinehart). His proposal of a media art notation
system (MANS), based on the MPEG-21 framework, is a
welcome step forward in the development of a viable
preservation schema for these highly variable and ephemeral
objects. Rinehart's system, however, is more of a metadata
framework or ontology than a scoring system, and therefore
runs into the same problems inherent in any text-based
representational framework that attempts to describe or define
a non-textual entity (Svenonius), and is based on the
‘conduit-metaphor' model of communication (Davis).
Musical Scores
There are numerous examples of existing art forms that
employ re-interpretation as the de facto means of delivery
and representation: namely; drama, for which scripts represent
the work; dance, which often makes use of a notational system
to describe and record body movements across space and time
(Labanotation being the primary example); and music, which
is usually represented in the West using the common notation
system (CNS). In order to develop a less text-centric
representational model for new media art, I chose to explore
the representation and interaction techniques of these art forms,
which have interpretable or variable output.
Music, drama and dance each share characteristics with new
media art. They're temporal: meaning they take place and
change over time; they're performance based and ephemeral:
the performance is the instantiation of the work, and once that
performance is over, it's gone; and they're open to interpretation
within some pre-determined set of values: although it is possible
and expected to interpret freely, each form has a framework
within which the director /choreographer/conductor must work.
However, music has traits uniquely shared with new media art.
For example, music and new media art both share a 'unity of
artistic vision', basically meaning that all instruments, in the
case of an orchestra; and libraries, programs, data sources, in
the case of a new media art object, interact in a specific way to
produce the final product. Another shared trait between music
and new media art is the existence of a level of abstraction that
is not necessarily present in drama or dance. Both the composer
and the programmer use instruments, or tools, to achieve his
or her artistic vision, whereas a playwright or choreographer
works with the more immediately available bodies in motion
and/or words as their means of expression. Finally, a play or
dance does not inherently depend on the availability of tools
to exist, whereas an orchestral / new media art performance
most often does.
In the hopes of developing a set of characteristics to include in
a new media art notation system, I have started exploring the
information contained in musical scores. In addition to
recording the fixed musical elements like pitch, rhythm, tempo,
dynamics, and articulation; this research also seeks to
understand musicians' interpretative decisions, as well as their
interactions with each other, and with the score. Interpretation
and interaction are particularly interesting for the purpose of
this research, because these are the primary means by which
musicians achieve artistic, reliable performances, and that is
the ultimate goal of any new media representation framework.
Although it might be difficult to completely understand
musicians' interpretative choices and interactions, we believe
that the personal notes (annotations) musicians' make on the
scores themselves can provide valuable information regarding
these transient and personal decisions (Marshall).
Research questions
Some of the questions this phase of the research seeks to
answer: Which musical elements must be regulated, which
can be improvised, and which must be freely interpreted? Is this dependant on context of presentation? Can any of these
musical elements transfer usefully to representation of new
media art? Under what circumstances does a musician decide
to do something that is not in the published score? How do
composers communicate over time, space, and cultures, their
intent and goals regarding performance? Are there different
models of interpretation based on different musical styles or
genres? How do composers, conductors, and musicians react
to the idea of official representation of interpretation?
Methodology
With these issues in mind, we developed an experimental
framework consisting of a musician/score collection
methodology; a coding schema, which will help categorize the
annotations; and a method for the systematic exploration of
annotations.
The musician/score hierarchy defines from whom and what
kinds of musical scores this study will consider. It is a structural
arrangement of four parallel interests and skills of each
musician: first is the three 'levels' of music-makers: musician,
conductor, and composer; second is the level of proficiency:
amateur, college-level, and professional; third, we take into
consideration the presence of a conductor: orchestras versus
quartets, for example; and finally, the hierarchy includes a
consideration of style of music – jazz, classical, and musical
theater.
After collecting the scores, we are marking up any annotations
the musicians/composers/conductors might have made on them.
There are two types of mark-up: structural, and content-based.
At the structural level, we decided to mark up at the bar level,
delineating the extent of bars and phrases, where appropriate.
At the content level, there are three types of written notes:
textual, where the musician has actually written a word in the
margins ( "Less Bow!!!" or "FROG!" ); symbolic, where the
musician has written non-textual symbols (stars, exclamation
points, and glasses); and numeric, where the musician has put
numbers above or below notes for fingering or timing
instructions, or numbered the bars if that information isn't
included in the published score.
At this point in the process we conduct interviews with the
participating musicians, asking questions regarding the process,
and context of annotation behavior (MacMullen). The purpose
of the interview is to get a deeper understanding of musicians'
attitudes toward interpretations and whether their annotation
behavior is in fact an important element in understanding that
interactive quality of musical performance.
The final step in this process is to analyze the annotated scores,
looking for n-way consensus, investigating any 'important' or
consistently documented sections or elements within a piece.
I will normalize at the basic unit of annotation (in this case at
the bar and phrase level); I'll record all instances of annotation:
who, where, what kind; and all count all instances of annotation
to provide percentages at the bar and phrase level between and
among musicians and musical types. Finally, I'll conduct
consensus analysis to determine how often annotations concur
on selections.
Initial Findings
After concluding a pilot study, exploring the initial data
from a university-level orchestra, and interviewing
several musicians and conductors, initial findings indicate that
the premises upon which this study are based are valid.
Annotation of the score provides insight into a number of
different issues relevant to the development of a notation
schema for new media art: annotation analysis identifies those
characteristics of musical notation important across musician
types and skill level, allowing me to make recommendations
for the inclusion of certain characteristics in the new notation
system. Annotation also evidences performance-related
interaction between and among musicians. Finally, annotation
is a physical sign of an individual musician's interaction with,
and interpretation of, the score itself.
This work is ongoing. Data collection will end in April 2005,
when more comprehensive findings will be available.
Acknowledgments
This work was partially funded by an unrestricted research
gift from Microsoft Research to the Annotation of
Structured Data research team in the School of Information
and Library Science at the University of North Carolina at
Chapel Hill, whose members contributed to this work: Gary
Marchionini, Paul Solomon, and Catherine Blake, co-PIs; with
team members Tom Ciszek, Xin Fu, Lili Luo, W. John
MacMullen, Cathy Marshall, Mary Ruvane, and David West.
The project's website is available at: <http://ils.unc.
edu/annotation/> .
Bibliography
Davis, Marc. "Theoretical foundations for experiential systems
design." Paper presented at ACM SIGMM Workshop on
Experiential Telepresence (ETP) 2003. New York: ACM Press,
2003. Depocas, Alain, Jon Ippolito, and Caitlin Jones. Permanence
through change: The variable media approach. New York,
NY: Guggenheim Museum Publications, 2003.
MacMullen, W.J. "Annotation as Process, Thing, and
Knowledge: Multi-domain studies of structured data
annotation." Paper delivered at ASIST Annual Meeting,
Charlotte, NC (October 28 - November 2, 2005). In Review.
Marshall, Catherine C. "Toward an ecology of hypertext
annotation." Paper presented at Hypertext98, Pittsburgh, PA,
1998. New York: ACM Press, 1998.
Rinehart, Richard. "A system of formal notation for scoring
works of digital and variable media art." Paper presented at
AIC Digital Media Group 2004 Annual Meeting, (Portland,
Oregon: June 14, 2004). 2004.
Svenonius, Elaine. "Access to nonbook materials: The limits
of subject indexing for visual and aural languages." Journal
of the American Society of Information Science and Technology
45.8 (1994): 600-606.