Multimedia Markup Editor (M3): A Semi-automatic Annotation Software for Static Image-Text Media

Introduction
This poster introduces an editor software specifically designed for graphic narratives, including graphic novels and comics, but also other kinds of illustrated still-image media. Users are able to mark up these documents in XML via a Java-based GUI. The annotation language used in the system, which we call “Graphic Novel Markup Language” (GNML), is an extension of the TEI-based “Comic Book Markup Language” (Walsh). Figure 1 presents an overview of the object types and relationships that have been implemented so far.

Fig. 1: Overview of object types in GNML

Features
A number of automatic processes in the M3 editor facilitate a simple annotation process. The system localizes outlines of panels by using processes used in computer graphics, such as the marching squares algorithm. Annotators may select simple structures such as speech bubbles or captions with a single click on element backgrounds, thus prompting a flood-fill process that recognizes the entire structure. For more complex representations such as characters, the system interprets annotators’ selections with the help of livewire segmentation (Mortensen & Barrett) to identify high-contrast edges and, subsequently, contours of characters. In order to mitigate the error rate of manual transcription, the system has a built-in spell checker and autocomplete function for text and character names.
One of the main goals of the annotation system is quantitative analysis, which aims to provide empirical evidence of narratological terminology, establish character networks, or provide access to comics text in the absence of automatic recognition. In the case of the former, a narratological annotation scheme that includes mark-up options for focalization and story worlds has been integrated into GNML. Theories of cognitive and transmedial narratology as well as empirical data (gathered with the help of the annotation system and subsequently analyzed) serve as a basis for this scheme. Figure 2 shows the annotation system GUI and a sample annotation.

Fig. 2: GUI and sample annotation of "Pepper & Carrot" (Revoy)

Conclusion & Outlook
The editor software facilitates the analysis of multimodal corpora with complex text-image interactions. Such evidence-based investigation may help revise existing theories of graphic narrative or falsify more qualitative scholarship. The software can also be used in classrooms for hands-on case studies and has been tested successfully in several university seminars. We are currently working on OCR integration, which presents particular challenges as graphic narratives often use handwritten or pseudo-handwritten fonts and existing OCR/ATR systems do not as yet provide satisfying results. We hope to integrate OCR either directly or as an optional step prior to manual annotation in future versions. The annotations system’s latest version can be downloaded at:

http://graphic-

literature.upb.de/?page_id=3592
. An FAQ to help annotators is available at:

http://graphic-

literature.upb.de/?page_id=4123

.
An open-source version will be made available by February 2020.

Bibliography
Dunst,
A.,
Hartel,
R.,
&
Laubrock,
J.
(2017).
The
Graphic
Narrative
Corpus
(GNC):
Design,
Annotation, and
Analysis
for
the
Digital
Humanities.

2nd
International
Workshop on
coMics
Analysis, Processing, and Understanding, 14th IAPR
International
Conference
on Document Analysis and Recognition.
Kyoto,
Japan.

Mortensen,
E.,
&
Barrett,
W.
(1995).
Intelligent
scissors
for
image
composition.

Proceedings of the 22nd annual
conference
on Computer graphics and interactive
techniques
(pp.
191-198).
ACM.

Revoy,
D.
(2017).

Pepper
&
Carrot
.
Episode
21: The
Magic
Contest
.
(Pepper
and
Carrot)
Retrieved September
10,
2018
from
https://
www.peppercarrot.com/en/article400/episode-21-the- magic-contest

Walsh,
J.
A.
(2012).
"Comic
book
markup
language:
An
introduction
and
rationale."

Digital Humanities Quarterly
, 6
(1).