The (Digital) Space Between: Notes on Art History and Machine Vision Learning

Machine vision learning and art historical practice are often poised as operations that are antithetical to one another (Spratt and Elgammal, 2014a, 2014b). A frequent criticism leveled by art historians against machine learning algorithms is that they do little that a trained art historian cannot do already (Bishop, 2017). A second criticism is that the results gleaned from machine vision learning are heuristic exaggerations. And a third criticism is that computer scientists simply do not understand how to approach visual art, and in the process wrongly (albeit unintentionally) define the field of art history for a much larger audience than the one the humanities tend to generate. Much of the value placed on machine vision learning as it pertains to understanding artworks has been on its ability to sort, classify, and match images with similar ones through style and genre (Saleh and Elgammal, 2015, 2016), taxonomies that fail to reflect the current state of the history of art.
The above criticisms present a fair critique of the approach of machine vision learning to the history of art – but only to a certain degree. Such criticisms fail to recognize that art historians are precisely the ones who have the greatest stake in and the greatest potential for contributing to the questions raised by machine learning image analysis. Art historians simply ask different questions about artworks – questions of history, scale, tactility, surface, and representation – than the ones of which computer scientists are aware. One reason for this disjuncture is that art historians have often kept to themselves instead of engaging with other disciplines that are intensely interested in visual imagery.
Rather than simply critiquing and lamenting how computer and data scientists approach visual imagery, this short paper addresses a few “between points”, as I call them, rather than intersections, where art historians can bring much critical insight into machine vision learning. For example, the issue of texture is a complex question in painting, for it can signify the texture of the paint, or the texture of the canvas weave, or how textured paint application is used to represent different physical textures, such as silk or fur. How could these distinctions be brought into machine vision learning? Another issue would be to see if a machine could identify when a painting was re-touched or repaired. Or one might compare how the descriptive terms generated by machine vision learning output correlate to the terms art historians would use when describing an object. The purpose of this paper is ultimately to pose some questions about how art historians and computer scientists might create a better dialogue in their respective practices.

Bibliography

Bishop, C. (2017). Against Digital Art History. https://humanitiesfutures.org/papers/digital-art-history/.

Spratt, E. and Elgammal, A. (2014a). Computational Beauty: Aesthetic Judgment at the Intersection of Art and Science. In
Agapito, L. et al (eds),

Computer Vision – ECCV 2014 Workshops. ECCV 2014. Lecture Notes in Computer Science

 8925. Cham: Springer International Publishing, pp. 35–53.

Spratt, E. and Elgammal, A. (2014b). The Digital Humanities Unveiled: Perceptions Held by Art Historians and Computer Scientists about Computer Vision Technology.
Cornell University Library arXiv Physics Archive. http://arxiv-web3.library.cornell.edu/abs/1411.6714v1.

Saleh, B. and Elgammal, A. (2016). Large-scale Classification of Fine-Art Paintings: Learning the Right Metric on the Right Feature.
International Journal of Digital Art History, 2: 71-94.