Computer Vision Tagging the Metropolitan Museum of Art's Collection

Abstract

Computer vision algorithms are increasingly being applied to museum collections to identify patterns, colors, and subjects by generating tags for each object image. There are multiple off-the-shelf systems that offer an accessible and rapid way to undertake this process. Based on the highlights of the Metropolitan Museum of Art's collection, this article examines the similarities and differences between the tags generated by three well-known computer vision systems (Google Cloud Vision, Amazon Rekognition, and IBM Watson). The results provide insights into the characteristics of these taxonomies in terms of the volume of tags generated for each object, their diversity, typology, and accuracy. In consequence, this article discusses the need for museums to define their own subject tagging strategy and selection criteria of computer vision tools based on their type of collection and tags needed to complement their metadata.

What do they make us see: a comparative study of cultural bias in online databases of two large museums

“…Researchers and educational institutions leverage the unique diversity and expertise of annotated datasets provided by major Western museums, such as the Metropolitan Museum of Art and the Rijksmuseum. Their online collections and API keys are used to study how algorithms can learn to identify various objects (Villeaespasa and Crider, 2021), and materials in paintings (Van Zuijlen et al. , 2021), generate art (Cetinik and She, 2021) and train students in data analysis and visualizations (Korman and Kessler, 2020; Choi, 2020).…”

“…The first is the museum's API dataset, which can be accessed via an API or downloaded to a personal computer, enabling users to search and analyze the information in the database directly. This type of database is mainly intended for commercial and academic parties that can study and research the information in it or use it to improve computational models (Tallon, 2018;Van Zuijlen et al, 2021;Villaespesa and Crider, 2021). A second type of database is accessible to end users through the museum's online search system, i.e.…”

“…This type of database is mainly intended for commercial and academic parties that can study and research the information in it or use it to improve computational models (Tallon, 2018; Van Zuijlen et al. , 2021; Villaespesa and Crider, 2021). A second type of database is accessible to end users through the museum's online search system, i.e.…”