Not Exactly Prima Facie: Understanding the Representation of the Human Through the Analysis of Faces in World Painting

In his 1872 book The Expression of Emotions in Man and Animals (Darwin 1872), Charles Darwin drew our attention to the relationship between human expressions, movements and emotional states, and tried to frame his conclusions by highlighting the similarities between humans and animals. The light he shed on cultural differences with respect to the appearance of the face and variations in expressions perceived across different groups to communicate the same emotions is also important.
More recent scientific evidence highlights the importance that the human beings give to the face. The brain has a specialized amygdala to discriminate scenes in favor of facial expressions, a primitive mechanism to detect potentially dangerous situations (Hariri et al. 2002), that explains our impulse to immediately look at people faces and try to read their facial expression. Finally, the recent discovery of “mirror neurons” (Rizzolatti and Craighero 2004) and their connection with the imitative ability of several primates offers a glimpse about the social construction of emotions, helps to explain the spread of behaviors within human groups, and opens up the possibility of a phenomenology of human expressions.
Nevertheless, the fascination with the human face is not something new in humanities. It has been present since the beginnings of art history; artists have always sought to relate the face to the human body (Chase 2005) and, especially, to the different ways in which how faces reflect the human condition. In this sense, human facial representations contain a human expressions and emotions archive that can help us understand, through a science of the face (Cleese and Ekman 2001), various human condition traits that evolved through time and space. We aim to answer questions about periods in art history, such as the Baroque significance as a culture derived from human expansion. Even if we cannot say that Baroque is just as a historical period, before the discovery of America all faces in art were mostly european; the presence of indigenous faces in paintings is a big disrupt. Another example would be the cultural meaning of the progressive human face disappearing from modern painting. Our methodology analyzes this through facial recognition techniques, data mining, graph theory and visualization and cultural history.
Quantitative analysis of huge amounts of data has provided answers to new and different questions that otherwise couldn’t have been considered (Michel et al. 2011). The study borrows some ideas from the Culturomics (Michel et al. 2011) concept by creating a set of more than 123,500 paintings from all periods of art history, and applying a face recognition algorithm used in Facebook’s photo-tagging system#. The result is a set of over 26,000 faces ready to be analyzed according to several features extracted by the algorithm.
The extracted information accuracy may fluctuate depending on several factors, such as the reproduction quality and size; the thematic content, a portrait is not the same than a hunting scene; the pictorial style, e.g., Cubism versus Figurative art; the lighting, dark and hellish overtones as opposed to daytime or celestial images; or even the contrast between the background and skin tone. At the same time, these elements also provide information about how human faces are depicted by authors, styles and cultures across time. Once a face has been recognized, it provides data about the position of eyes, mouth, chin and ears — what we call the face basic features. If the confidence measure we have reached during the recognition process is satisfactory, we can consider gender, mood, position of lips, range of age or even if the person is wearing glasses — we call these extended features of facial recognition.

Figure 1:
Example of face graph for two randomly selected paintings. The red crosses indicate what we called face basic features. The blue lines show the different distances in between the features.
The methodology tackles the study of this huge amount of features in three steps. First, we build a graph with the set of basic features. Second, we look for clusters in the extended features. Third, we compare the graphs and the clusters, corresponding to the basic and extended features respectively, using time intervals and space as factors of the comparison. The selection of the procedures in each of the steps considers different strategies to address the fluctuations in accuracy. What follows is a more detailed explanation of the main mathematical and computational tools used for the processing of the data in each step.
For the basic features graph (Figure 1), we apply the Elastic Bunch Graph Matching (Wiskott et al. 1997) to the extracted data, and then we calculate the resulting weighted graph after applying graph similarity functions like Euclidean Geometry Similarity and Least Square Geometry Similarity. Once the graphs that represent the 26,000 faces are done, we normalize the weights and prune those relationships with weights below the first quartile. Then, a combination of YiFan Hu Multilevel (Hu 2005) and ForceAtlas (Bastian et al. 2009) layouts algorithms are run against the graph to see how the faces are clustered according to their modularity class.
For the extended features clustering, we create a vector for each face and run the K-Means method (Sculley 2010) that is able to cluster unlabeled data, i.e there is no need of a pre-training process but the classification emerges naturally from the data similarities. We check the clusters obtained according to their homogeneity and completeness, according to the definitions given by Rosenberg and Hirschberg (Rosenberg and Hirschberg 2007), in order to establish a good value for V-measure, a conditional entropy-based external cluster evaluation measure that indicates the success of a clustering solution.
Finally, we compare the two sets: the basic features set using graphs and the extended features set using clustering by K-Means method (Sculley 2010). At this point, we are at the perfect position to analyze and characterize each of the groups according to different historical perspectives and cultural questions, for instance, the distinction among styles by giving a minimum set of features that determines its membership. A similar set of features is obtained for a particular interval of time or a specific geographical region. Extending that analysis we are able to study the evolution of these sets of features over time.
In this study, we tackle three important issues that also show the potential of this kind of work to develop new approaches to cultural history and to establish some yardsticks in order to complement the incipient methodology for a Big History (Christian 2004) approach to human culture. First, we show how the analysis of the representation of human faces — both the internal features and in its relative position to the rest of the composition — offers important data to determine periods and borders in the history of art beyond the generalizations supported by the notions of “style”, “genre” and “national history”. Second, we study the correlations between the European expansion overseas from the 16th Century onwards, and the introduction of new human “types” in world paintings, focusing on concepts of identity and gender (with special emphasis on the size and form of the forehead), and relating the results to notions of Baroque, hybridization and globalization (Suarez 2007; Suarez 2008). Finally, we move to the 20th Century and study the disappearance of the human face from art in relation to Ortega y Gasset’s concept of the “dehumanization of art” (Ortega y Gasset 1968) and the artistic and political movements of the first half of the century.
References
Bastian M., S. Heymann, and M. Jacomy (2009). Gephi: an open source software for exploring and manipulating networks. International AAAI Conference on Weblogs and Social Media.
Chase, P. G. (2005). The Emergence of Culture: The Evolution of a Uniquely Human Way of Life. Birkhäuser.
Christian, D. (2004). Maps of time: an introduction to big history. Heldref Publications.
Cleese, J., and P. Ekman. (2001) The Human Face. [BBC] John Cleese, Paul Ekman. http://www.bbc.co.uk/programmes/b00pfqy6
Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.
Hariri, A. R., et al. (2002). The Amygdala Response to Emotional Stimuli: A Comparison of Faces and Scenes.NeuroImage 17: 317–323.
Hu, Y. (2005). Efficient, high-quality force-directed graph drawing. Mathematica Journal. mathematica-journal.com.
Michel, J. B., et al. (2011). Quantitative Analysis of Culture Using Millions of Digitized Books. Science 331(6014): 176-182.
Ortega y Gasset, J. (1968). The Dehumanization of Art and Other Essays on Art, Culture, and Literature. Princeton Paperbacks. 128.
Press notice: https://developers.face.com/signup/?g and http://face.com/blog/facebook-acquires-face-com/ Accessed October, 31st , 2012.
Rizzolatti, G., and L. Craighero, (2004). The mirror-neuron system. Annual Revision Neuroscience.annualreviews.org
Rosenberg, A., and J. Hirschberg. (2007). V-measure: A conditional entropy-based external cluster evaluation measure'. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning held June 2007 in Prague. 410–420.
Sculley, D. (2010). 'Web Scale K-Means clustering'. Proceedings of the 19th international conference on World wide web.
Suarez, J. L. (2007). Hispanic Baroque: A Model for the Study of Cultural Complexity in the Atlantic World. South Atlantic Review 72(1): 31-47.
Suarez, J. L. (2008). Complejidad y Barroco. Revista de Occidente. 323. 58-74.
Wiskott, L., et al. (1997). 'Face recognition by elastic bunch graph matching. Pattern Analysis and Machine Intelligence'. IEEE Transactions. 19(7): 775-779.