Fairness in Proprietary Image Tagging Algorithms: A Cross-Platform Audit on People Images

Abstract

There are increasing expectations that algorithms should behave in a manner that is socially just. We consider the case of image tagging APIs and their interpretations of people images. Image taggers have become indispensable in our information ecosystem, facilitating new modes of visual communication and sharing. Recently, they have become widely available as Cognitive Services. But while tagging APIs offer developers an inexpensive and convenient means to add functionality to their creations, most are opaque and proprietary. Through a cross-platform comparison of six taggers, we show that behaviors differ significantly. While some offer more interpretation on images, they may exhibit less fairness toward the depicted persons, by misuse of gender-related tags and/or making judgments on a person’s physical appearance. We also discuss the difficulties of studying fairness in situations where algorithmic systems cannot be benchmarked against a ground truth.

Shifting Our Awareness, Taking Back Tags: Temporal Changes in Computer Vision Services’ Social Behaviors

“…We replicated the data collection (Barlas et al 2019) and analysis (Kyriakou et al 2019) followed in October 2018, this time in August 2021. We compared the two analyses, aiming to uncover the differences in the behaviors of the ITAs after almost three years.…”

“…For our 2018 audit, we manually constructed a typology of tags which allowed us to compare the taggers in their use of concepts, despite their differing vocabularies. While our paper (Kyriakou et al 2019) accompanying dataset (Barlas et al 2019) have more information, we briefly describe the typology here. The typology consists of four super-clusters, which in turn contain a total of 15 sub-clusters.…”

“…Likewise, while the performance of facial recognition algorithms has improved over time, they still tend to perform best on images depicting white men. 2 Inspired by Gender Shades, others have investigated the performance disparities for race, gender and age in computer vision during the past years, highlighting higher error rates for specific race/gender identities (Kyriakou et al 2019), skin colors (Raji and Buolamwini 2019), and age groups (Phillips et al 2011).…”

Adaptive Clustering of Robust Semantic Representations for Adversarial Image Purification on Social Networks

“…In (Moreira et al 2016), authors introduced a non-deep learning based, temporally robust feature extractor and a bag of visual words method to classify pornographic videos with considerable accuracy. Content modified images, which skew perceptions of viewers (Reis et al 2020), and unfair classifiers, which are biased to different target populations (Kyriakou et al 2019) are also topics concerning social media image analytics.…”

Fairness in Algorithmic and Crowd-Generated Descriptions of People Images

“…In this talk, I will discuss recent work in analyzing proprietary image tagging services (e.g., Clarifai, Google Vision, Amazon Rekognition) for their gender and racial biases when tagging images depicting people. I will present our techniques for discrimination discovery in this domain [2], as well as our work on understanding user perceptions of fairness [1]. Finally, I will explore the sources of such biases, by comparing human versus machine descriptions of the same people images [3].…”