From Mnemosyne To Terpsichore – The Bilderatlas After The Image

Introduction
This study concerns Aby Warburg's last and most ambitious project: the Atlas Mnemosyne (or Bilderatlas), conceived in August 1926 and truncated three years later, unfinished, by Warburg's sudden death in October 1929. Mnemosyne consists of a series of large black panels, on which are attached a variable number of black-and-white photographs of paintings, sculptures, tarot cards, stamps, coins, and other types of images. The version we use is the one Warburg was working on at the time of his death, also known as the “1-79 version”: it includes around a thousand images

pinned on 63 panels (This version of the Atlas is published in various print editions, and available online). The Bilderatlas is a conceptual maze - the culmination of a life's scholarship in images and memory - through which perhaps the clearest thread is the concept of Pathosformel, or formula for (the expression of) passion. Much excellent work has been written on the concept but, to the best of our knowledge, nobody has yet tried to “operationalise” it - to turn it into a sequence of quantitative operations, or in other words, into an algorithm (Moretti, 2013).

The Pathosformel and its Operationalisation
On the most basic level, the Pathosformel describes the portrayal passionate emotions through a formula, a repeatable visual paradigm. The Pathosformel owes much of its force, as Salvatore Settis has pointed out, to its combination of semantic opposites: an “oxymo-ronic word, in that it merges in the same term the movement of pathos and the rigidity of the formulaschema” (Settis 1997).

Rather than attempting to operationalise the entire concept at once, we first break it down into its constituent components: first the morphology of formula, then the dynamism of pathos.

We turn to two well-studied cases of Warburgian formula: the Death of Orpheus (where Warburg first named the concept of Pathosformel), and the Nymph, headhuntress and Fortuna (Bilderatlas panels 46-48). Looking at Warburg's examples of the Orpheus-formula, reproduced in Figure 1, we can hardly stop ourselves from spotting a formula which repeats across the ages; but how could this intuitive similarity be measured?

Our algorithm comes as follows:

1. We isolate each individual body from its context. It is clear the Pathosformel relates to individual characters - the object of study thus becomes not panels or photographs in the Bilderatlas, but individual bodies.

2. We take only the skeletons of such bodies. Here we are eliminating colour, clothes, hands, faces, gender, age. This is not to say that such factors aren't important: but they are not elementary to the formula (see for example, André Jolles' letter to Warburg 23rd December 1900, where the formula of Ghirlandaio's Nymph hops

between Judith, Salome, Tobias, Gabriel

etc.- see Ghelardi, 2004; or the formula in Bilderatlas Panel 47 shared between Giambolo-gna's Samson and Donatello's Judith.)

3. We compare these skeletons by measuring the angles of the main limbs of the body, as described in Section 5.

Each of these steps is not merely a convenient quantification, but a conceptual wager. This is the strength of operationalisation as a critical tool: it forces one to be explicit about the conceptual choices one makes.

Fig 1: the Death of Orpheus: details from Warburg's example figures from classical antiquity to Dürer.

Reproduced from Warburg 1905/1999

Anatomy, Emotion and Pose
Warburg was certainly influenced by Darwin's The Expression of the Emotions in Man and Animals (Darwin 1872) - which, when talking about human emotions, largely concentrates on the face. Indeed, the first figure of the book is an anatomical diagram of the face - “I shall often have to refer [...] to the muscles of the human face” (Darwin 1872 p.23). Warburg was certainly struck by the book - writing in his diary “ein Buch, das mir hilft!” (Gombrich, p.72). He was also interested and capable of studying the face in art (see e.g. his discussion of faces in Ghirlandaio's Confirmation in Santa Trinita, in The Art of Portraiture in the Florentine Bourgeoisie (in Warburg 1999 p.185), yet never in relation to Pathos - his descriptions of the Pathosformel relate exclusively to the body.

We can relate Warburg's decision to the large psychological literature on emotional recognition from bodies. Psychological studies are based on the Light Spots Model by Johansson (1973), often called ‘biological motion', in which reducing body pose to 10-12 points - quite comparable to our own reduction - is judged to give a ‘compelling impression of human walking, running, dancing etc.'. Using only Light Spots, observers can reliably tell gender and emotion from dynamic pose (Kozlowski 1977, Montepare 1987). Indeed, it has been suggested that our emotional understanding of faces is more influenced by our perception of the body than vice versa (Van den Stock 2007).

Encoding Pathos through Pose
The Atlas is, even by today's dataset standards, quite sizeable: 1000 images across 63 panels, containing an order of 103-104 depicted human figures. Scalable manual annotation is only therefore possible through crowdsourcing, which we did through the CrowdFlower platform.

Accurately annotating every visible figure in an image is a difficult and ambiguous tasks. Additionally, if different workers annotate different figures in the same image, the annotations cannot be collated or averaged. We therefore developed a two-stage annotation process:

Human figures are extracted from the painting by drawing bounding-boxes. This is done three times per image (by three separate workers).

Having aggregated the information from the first stage, separate images are produced for each human figure. Detailed pose information (the position of major body-points) is then added by three separate workers, with the information aggregated.

It should be clarified that the decision to annotate bodies in isolation (for annotation accuracy and just worker compensation) is quite separate to the earlier conceptual decision to analyse bodies individually, which relates to the object of study. It would be quite possible to do either one without the other.

Using this two-stage annotation process, we have

presently annotated % of the Bilderatlas (by panels), resulting in 1,665 aggregated human poses. The collection and aggregation of the data are described in greater technical detail elsewhere (Impett, 2016).

Data Analysis: dimensionality reduction and dimensioned reductionism
Having encoded our static poses, how do we analyse and compare a collection of human figures of different sizes, proportions and orientations? We mirrored the poses horizontally and controlled for global rotation, ending up with a 11-dimensional vector P, describing the angles of the main limbs.

From this angular pose vector, we can use circular statistics to find a morphological distance Da,b between two poses Pa and Pb:

io

Da,b= £ \Pa<i il

i=l

Where Pa, i is the ith angle of pose vector Pa , and | |a is the angular radian distance:

|x| a = min(x, n - x)

These morphological pose-differences are perceptually meaningful over short distances. On a larger scale, they become less perceptually significant: is a sitting person ‘closer' to a lying or standing person?

In order to make our distance analysis perceptually relevant, therefore, we first clustered our 1,665 poses into 16 clusters by rotational K-means clustering (Dordet-Berdanet and Wicker 2008). Our two-stage clustering system is therefore as follows:

I. K-means clustering (to produce meaningful clusters)

II. Hierarchical clustering (for within-clus-ter morphological information)

The number of clusters K is chosen by looking at the inter-cluster variance over K. The result of the first stage of clustering is shown in Figure 2; Figure 3 shows an example section from a hierarchical map of Cluster 1.

Fig 2: our 16 pose-clusters

Fig 3: a detail from a dendrogram of different poses within the Bilderatlas, produced by second-stage clustering within

Cluster 2

Unity of the Pathosformeln: from distant to close reading
Some of the clusters clearly represent physical activities - sitting, praying, embracing, dancing - whilst others seem more subtly communicative or expressive in nature. Having reorganised the 1,665 figures into 16 mean-centred clusters, we proceeded to trace the classical Pathosformeln - identified in the Atlas by primary and secondary literature - through our clusters.

The canonical Pathosformeln are mainly mythological figures (Perseus, Pentheus, Orpheus) or recurring allegories (Graces, Nymphs, Fortuna). They were previously described as distinct, and we expected to find a taxonomy of such formulae through our clustering analysis.

On the contrary, the statistical result was much stronger: a complete morphological unity. Almost every identified Pathosformel falls into Cluster 1, with few false positives - over 80% of the figures in Cluster 1 are an identified Pathosformel. The handful of exceptions are all borderline cases, placed in peripheral to Cluster 1 (Clusters 7 and 13).

Looking more closely at the images themselves, as in Figure 4, this becomes visually clear: not only do the Pathosformeln share certain pose features (most importantly, a raised arm) present nowhere else in the dataset. To date, however, the authors know of no art-historical literature that has identified such morphological unity.

Figure 4, clockwise from top-left: Laocoon, Orpheus, Fortuna, Nymph, Judith, Perseus - all except Orpheus are in the Atlas. The identified Pathosformeln share distinguishing features from the other characters in the Atlas: a raised arm,

most often accompanied by a lowered second arm, and a slight twist of the body.

Concluding remarks
Our morphological model for Pathosformel is statistically strong: but what are the art-historical implications? The oppositional symmetry and raised arm of Cluster 1 (Fig. 2) reminds us of a Contrapposto, but the bodies themselves are far removed from such classical balance (e.g. Fig. 4, top). Rather than movement, tension (between upper and lower body) seems to be the fundamental element of Pathosformel - the nature of which will be the subject of a subsequent publication.

Our morphological analysis has shown that static pose can identify Pathosformeln, and that a study of static pose through a large collection of artistic works can identify links across styles, periods and cultures.

Automatic detection of pose is a focus of the current research, and will allow us to expand our art-historical models beyond the Bilderatlas. Humans (thus bodies, and poses) are unsurprisingly the most common feature of human art, and therefore make excellent objects of study for an art history of the Longue Durée (Robb 2015).

Warburg, A., trans. D. Britt. (1999) The Renewal of Pagan Antiquity. Los Angeles, Getty Research Institute.

Bibliography
Darwin, C. (1872). The expression of the emotions in man and animals. London, John Murray

Dortet-Bernadet, J.L. and Wicker, N., (2008). Model-based clustering on the unit sphere with an illustration using gene expression profiles. In Biostatistics, 9(1), pp.6680.

Gombrich, E.H. (1970). Aby Warburg, an intellectual biography. Chicago University Press.

Impett, L. and Süsstrunk, S., (2016). Pose and Pathosformel in Aby Warburg's Bilderatlas. In European Conference on Computer Vision (pp. 888-902). Springer International Publishing.

Johansson, G., (1973). Visual perception of biological motion and a model for its analysis. In Perception & psychophysics, 14(2), pp.201-211.

Kozlowski, L.T. and Cutting, J.E., (1977). Recognizing the sex of a walker from a dynamic point-light display. In Perception & Psychophysics, 21(6), pp.575-580.

Montepare, J.M., Goldstein, S.B. and Clausen, A., (1987). The identification of emotions from gait information. In Journal of Nonverbal Behavior, 11(1), pp.33-42.

Moretti, F., (2013). “Operationalizing”; or, The Function of Measurement in Modern Literary. Stanford Literary Lab Pamphlet Series. litlab.stanford. edu/Liter-aryLabPamphlet6

Robb, J. and Harris, O.J.T. (2013). “Body worlds and their history: some working concepts.” The Body in History: Europe from the Palaeolithic to the future: 7(31).

Settis, “Pathos und Ethos, Morphologie und Funktion”, in W. Kemp, G. Mattenklott, M. Wagner, M. Warnke, eds, Vorträge aus dem Warburg-Haus, Band 1, De Gruyter, Berlin 1997, pp. 39-44.

Van den Stock, J., Righart, R. and De Gelder, B., (2007). Body expressions influence recognition of emotions in the face and voice. In Emotion, 7(3), p.487.