A machine learning methodology to analyze 3D digital models of cultural heritage objects

Thanks to recent advances in scanning technologies there has been an increase in the number of methods developed for digitizing cultural heritage objects. Many of the resulting 3D models are used for visualization or archiving purposes. Unfortunately, there are still few projects oriented to gain archaeological knowledge from point clouds and triangular meshes.
In this paper we present some results of an ongoing project that applies machine learning and computer vision techniques for recognizing, retrieving and classifying cultural heritage objects in an automatic way (Jiménez-Badillo, et al. 2010, 2013; Jiménez-Badillo and Román-Rangel, 2016, 2017; Roman Rangel and Jiménez-Badillo, 2015, Román-Rangel et al., 2014, 2016a, 2016b; Jiménez-Badillo and Ruiz-Correa, 2017). The presentation focuses specifically on a method to analyze style variations of archaeological artefacts with minimal human intervention. This is based on a 3D morphing algorithm proposed by Shelton (2000). Our implementation allows analyzing pairs of objects whose shapes represent the canonical extremes of a continuum, that is, objects that belong to two different “styles” within a cultural tradition. The purpose of the algorithm is taking two extreme shapes (i.e. 3D point-clouds, surface meshes or 3D digital models) as input in order to extract several 3D virtual models whose shape or “style” lies “in-between” the two extremes. This is useful in situations where archaeologists need to decide to which extreme a real artefact is more similar. Archaeologists can also apply the algorithm to compare each object of a collection against all the other members of the set. This would produce an “atlas” of the shape variations expected for such collection, which in turns would facilitate the application of a classification method based on machine learning.
The formal mathematical details of this approach can be found in the original paper by Shelton (2000). During the presentation, we plan to offer an intuitive explanation of the algorithm for the benefit of those Humanists who are not experts in mathematics. This can be summarized as follows:

Mathematically speaking, the problem consists in finding correspondences between two 3D point-clouds or surface meshes. This means finding points in surface “A” that match corresponding points in surface “B” with minimal user intervention.
The challenge is how to make that the algorithm recognizes meaningful geometric correspondences between models “A” and “B”. In other words, how to find structural geometric correspondences between the points that define, for example, the nose of model “A” with the nose of model “B”, and the same for all other features of the masks.
The solution proposed by Shelton is a mathematical equation that fits three criteria:

Similarity. For each point a on surface A, the function
C(a) must find a point close or on the surface B.

Structure. Function C must produce the least possible distortion in the transition from A to B. In other words, the result of function C(a) must have a geometric structure similar to A.

Plausibility. Function C must represent a realistic model derived from surface A.

The first property establishes that C must find real points in A that match points in B. The second condition establishes that the correspondences found between A and B must not be arbitrary. On the contrary, there must be matching substructures of A present in B (e.g. the matching of the nose in A must have some correspondence with the nose in B), so that the deformation makes sense (figure 1). The last condition guarantees that the deformation includes the previous knowledge of the user in terms of which forms are acceptable for the deformation, because it makes no sense to transform a face mask into an airplane, for example.
The idea for this project came from the need to rank shape similarities in a collection of archaeological stone masks from Mexico. This includes masks belonging to several well-defined styles, but it also includes many others that cannot be clearly positioned within a specific class because they share features of two or more canonic styles (figure 2).
These masks were found in the Sacred Precinct of Tenochtitlan, the main ceremonial Aztec complex, located in Mexico City. The schematic features of these objects set them apart from other artifacts with more naturalistic style. This has attracted the attention of many specialists and during the last three decades these items have been the subject of intense debate for two main reasons:
First, the 162 masks were located in 14 Aztec offerings dating from 1390 to 1469 A.C., yet they do not show typical “Aztec” features. Indeed, their appearance resembles artifacts from the southern State of Guerrero, particularly from the Mezcala region, which is hundreds of kilometers away from the ancient Tenochtitlan.
Such origin would not be rare, as it was common for the Aztecs to import goods from other regions either by trade or by extracting tribute from conquered towns. The style of the masks and figurines, however, is more difficult to explain. It is similar, if not identical, to the style of objects produced in Mezcala and other places of Guerrero during much earlier times, probably during Classic (200 to 1000 A.C.) or even Preclassic times (2000 B.C. to 200 A.C.), while the offerings are Late Postclassic contexts. This leads to the question: Did the Aztecs collected “antique” objects to re-use them in their own offerings?, or the Guerrero/Mezcala styles survived till the late Postclassic period and therefore the offering objects were produced during Aztecs times? It is worth noticing that before the Aztec offering findings very few Mezcala style artifacts had been found in Postclassic contexts. Unfortunately, not enough stratigraphc information is available for collections from Guerrero, so specialists rely purely on stylistic considerations to explain the chronology of these artifacts.
Second, it is not clear how many Guerrero/Mezcala styles exist. Some specialists believe there are at least five different traditions (Covarrubias, 1948, 1961; Olmedo and González, 1986; González and Olmedo, 1990), while others recognize only four (Gay, 1967) or two (Serra Puche, 1975). The diversity of views is due in part to a lack of contextual information available for the majority of artifacts found in Guerrero, but it also reflects the subjective criteria used to classify such artifacts.
Clearly, more objective methods are needed to answer questions such as: how many styles were developed in the Guerrero/Mezcala regions?; which specific styles are represented among the offering objects found in the Sacred Precinct of Tenochtitlan?; and more importantly for the purposes of this paper: To which style each mask belongs?
Previous studies have tried to solve some of these questions by analyzing object shapes with clustering methods (Olmedo and González, 1986 , González and Olmedo, 1990, Jiménez-Badillo and Ruiz-Correa, 2017), but we believe that the application of morphing algorithms could produce a more objective assessment to solve the problem of style attribution in this and other archaeological collections.
Our application takes examples of two canonical styles and applies the deformation algorithm in order to produce a hundred virtual 3D models whose shapes go from one to the other extreme (figure 3). The virtual models produced in this way represent intermediate steps from style “A” to “B”. Each virtual model has associated a number that indicates its degree of similarity to style “A” or “B”. We can then examine a real archaeological object “c” to determine if its shape is closest to “A” or “B” and by how much. During the presentation we demonstrate a piece of software that implements the morphing algorithm and show, in a visual way, which parts of a 3D model suffer more deformation while transitioning from style “A” to “B” (figure 4).
As this is a work in progress, we are interested in receiving feedback from the audience about the relevance of our tools to resolve similar or new archaeological questions and welcome collaboration with other research projects willing to try this generic software for new applications.

Figure 1. The morphing algorithm proposed by Shelton (2000) seeks to generate a sequence of intermediate virtual models from “A” to “B”. To do that, it needs to identify correspondences in geometric substructures (i.e. noses, mouths, etc.) in both models

Figure 2. The first three columns from left to right show nine masks belonging to three different styles from Guerrero, Mexico. The fourth column on the extreme right shows three masks that cannot easily be attributed to the Sultepec, Chonatl or Mezcala styles

Figure 3. A sequence of 3D virtual models produced with Shelton´s algorithm (Shelton, 2000). Notice that each model represents a transition between the shape of mask “A” and the shape of masks “B”

Figure. 4. A snapshot of the morphing software implemented as part of the research project

Bibliography

Covarrubias, M. (1948). Tipología de la industria de piedra tallada y pulida de la cuenca del Río Mezcala. In
El Occidente de México. México: Sociedad Mexicana de Antropología, pp. 86-90.

Covarrubias, M. (1961).
Arte indígena de México y Centroamérica. México: Universidad Nacional Autónoma de México.

Gay, C. T. (1967).
Mezcala Stone Sculpture: The Human Figure. New York: The Museum of Primitive Art, Studies Number Five.

González, C., y Olmedo, B. (1990).
Esculturas Mezcala en el Templo Mayor. México: Instituto Nacional de Antropología e Historia.

Jiménez-Badillo, D. and Román-Rangel, E. (2016). Application of the ´Bags-of-Words Model´ (BoW) for Analyzing Archaeological Potsherds. In Campana, S., Scopigno, R., Carpentiero, G., and Cirillo, M. (eds.),
CAA 2015. Keep the Revolution Going. Proceedings of the 43
rd
Annual Conference on Computer Applications and Quantitative Methods in Archaeology, Vol. 2. Oxford: Archaeopress, pp. 847-856..

Jiménez-Badillo, D. and Román-Rangel, E. (2017). Clasificación automática de fragmentos de vasijas arqueológicas mediante el modelo Bolsa de Palabras. In Jiménez-Badillo (ed.),
Arqueología Computacional. Nuevos enfoques para la documentación, análisis y difusión del patrimonio cultural. México: Instituto Nacional de Antropología e Historia, pp. 111-126.

Jiménez-Badillo, D. and Ruiz-Correa, S. (2017). Análisis tridimensional de objetos arqueológicos con técnicas de visión por computadora. In Matos Moctezuma, E. and Ledesma Bouchan, P. (eds.), T
emplo Mayor. Revolución y estabilidad. México: Instituto Nacional de Antropología e Historia, pp. 199-214.

Jiménez-Badillo, D., Ruíz-Correa, S. and García Alfaro, W. (2010). 3D Shape Matching and Retrieval for Archaeological Analysis. In Melero. F. J., Cano, P. and Revelles, J. (eds.),
Fusion of Cultures. Abstracts of the 38
th
Annual Conference on Computer Applications and Quantitative Methods in Archaeology, Granada, Spain, April 6-9, 2010, pp. 583-586.

Jiménez-Badillo, D., Ruiz-Correa, S. and García Alfaro, W. (2013). Developing a Recognition System for the Retrieval of Archaeological 3D Models. In Contreras, F., Farjas, M. and Javier Melero, F. J. (eds.),
CAA Fusion of Cultures. Proceedings of the 38
th
Annual Conference on Computer Applications and Quantitative Methods in Archaeology, Granada, Spain, April 6-9, 2010. Oxford: Archaeopress, BAR International Series 2494, pp. 325-332.

Olmedo, B., y González, C. (1986).
Presencia del estilo Mezcala en el Templo Mayor: Una clasificación de piezas antropomorfas. Thesis presented as requirement for the degree of Bachelor in Archaeology. México: Escuela Nacional de Antropología e Historia.

Román-Rangel, E. and Jiménez-Badillo, D. (2015). Similarity Analysis of Archaeological Potsherds Using 3D Surfaces. In Carrasco-Ochoa, J. A. et al. (eds.),
Lecture Notes in Computer Science, Vol. 9116,
Proceedings of the 7
th
Mexican Conference on Pattern Recognition, Mexico City, June 24-27, 2015
. Switzerland: Springer International Publishing, pp. 125-134. DOI: 10.1007/978-3-319-19264-2 13

Roman-Rangel, E., Jimenez-Badillo, D. and Aguayo-Ortiz, E. (2014). Categorization of Aztec Potsherds Using 3D Local Descriptors. In Jawahar, C. V. and Shan, S. (eds.),
Lecture Notes in Computer Science, Vol. 9009:
Computer Vision - ACCV 2014 Workshops
, Part II, Singapore, November 1-2, 2014. Switzerland: Springer International Publishing, pp. 567-582. DOI: 10.1007/978-3-319-16631-5 42

Román-Rangel, E., Jiménez-Badillo, D., and Marchand-Maillet, S. (2016a). Classification and Retrieval of Archaeological Potsherds Using Histograms of Spherical Orientation.
ACM Journal of Computing and Cultural Heritage 9(3): 17:1-17:23.

Román-Rangel, E., Jiménez-Badillo, D., and Marchand-Maillet, S. (2016b). Rotation Invariant Local Shape Descriptors for Classification of Archaeological 3D Models. In Martínez Trinidad, J. F. et al. (eds.),
Lecture Notes in Computer Science, Vol. 9703:
Proceedings of the 8
th
Mexican Conference on Pattern Recognition, Guanajuato, Mexico, June 22-23 2016
. Switzerland: Springer International Publishing, pp. 13–22. DOI: 10.1007/978-3-319-39393-3 2

Serra Puche, M. C. (1975). Intento de seriación en esculturas de Guerrero. Cronología del estilo Mezcala. In
XIII Mesa Redonda de la Sociedad Mexicana de Antropología, Jalapa, México: Sociedad Mexicana de Antropología.

Shelton, C. R. (2000). Morphable Surface Models.
International Journal of Computer Vision, 38(1): 75-91.