Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Chen, Baiyu; Escalera, Sérgio; Guyon, Isabelle; Ponce-López, Víctor; Shah, Nihar B.; Simón, Marc Oliu

doi:10.1007/978-3-319-49409-8_33

Cited by 38 publications

(26 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feasibility of the challenge annotations was successfully evaluated prior to the start of the challenge. The reconstruction accuracy of all annotations obtained by the BTL model was greater than 0.65 (test accuracy of cardinal rating reconstruction by the model [15]). Furthermore, the apparent trait annotations were highly predictive of invite-for-interview annotations, with a significantly above-chance coefficient of determination of 0.91.…”

Section: Data Annotationmentioning

confidence: 81%

Modeling, Recognizing, and Explaining Apparent Personality From Videos

Kaya

Salah

Escalera

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Explainability and interpretability are two critical aspects of decision support systems. Within computer vision, they are critical in certain tasks related to human behavior analysis such as in health care applications. Despite their importance, it is only recently that researchers are starting to explore these aspects. This paper provides an introduction to explainability and interpretability in the context of computer vision with an emphasis on looking at people tasks. Specifically, we review and study those mechanisms in the context of first impressions analysis. To the best of our knowledge, this is the first effort in this direction. Additionally, we describe a challenge we organized on explainability in first impressions analysis from video. We analyze in detail the newly introduced data set, evaluation protocol, proposed solutions and summarize the results of the challenge. Finally, derived from our study, we outline research opportunities that we foresee will be decisive in the near future for the development of the explainable computer vision field.Keywords Explainable computer vision · First impressions · Personality analysis · Multimodal information · Algorithmic accountability 1 IntroductionLooking at People (LaP) -the field of research focused on the visual analysis of human behavior -has been a very active research field within computer vision in the last decade [28,29,62]. Initially, LaP focused on tasks associated with basic human behaviors that were obviously visual (e.g., basic gesture recognition [71,70] or face recognition in restricted scenarios [10,83]). Research progress in LaP has now led to models that can solve those initial tasks relatively easily [66,82]. Instead, attention on human behavior analysis has now turned to problems that are not visually evident to model / recognize [84,48,72]. For instance, consider the task of assessing personality traits from visual information [72]. Although there are methods that can estimate apparent personality traits with (relatively) acceptable performance, model recommendations by themselves are useless if the end user is not confident on the model's reasoning, as the primary use for such estimation is to understand bias in human assessors.Explainability and interpretability are thus critical features of decision support systems in some LaP tasks [26]. The former focuses on mechanisms that can tell what is the rationale behind the decision or recommendation made by

show abstract

Section: Data Annotationmentioning

confidence: 81%

Modeling, Recognizing, and Explaining Apparent Personality From Videos

Kaya

Salah

Escalera

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Amazon Mechanical Turk (AMT) was used for generating the labels. A principled procedure was adopted to guarantee the reliability of labels, converting rankings provided by labelers into normalized real valued scores (see [45] for details). The considered personality traits Fig.…”

Section: B Datamentioning

confidence: 99%

“…In addition to labeling the apparent personality traits, AMT workers labeled each video with a variable indicating whether the person should be invited or not to a job interview (the "job-interview variable"). This variable was also subject to the post processing reported in [45], so the variable to be predicted is a real number. The reader is referred to [18] where the data set is described in more details.…”

Section: B Datamentioning

confidence: 99%

Design of an explainable machine learning challenge for video interviews

Escalante

Guyon

Escalera³

et al. 2017

2017 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

International audienceThis paper reviews and discusses research advances on “explainable machine learning” in computer vision. We focus on a particular area of the “Looking at People” (LAP) thematic domain: first impressions and personality analysis. Our aim is to make the computational intelligence and computer vision communities aware of the importance of developing explanatory mechanisms for computer-assisted decision making applications, such as automating recruitment. Judgments based on personality traits are being made routinely by human resource departments to evaluate the candidates' capacity of social insertion and their potential of career growth. However, inferring personality traits and, in general, the process by which we humans form a first impression of people, is highly subjective and may be biased. Previous studies have demonstrated that learning machines can learn to mimic human decisions. In this paper, we go one step further and formulate the problem of explaining the decisions of the models as a means of identifying what visual aspects are important, understanding how they relate to decisions suggested, and possibly gaining insight into undesirable negative biases. We design a new challenge on explainability of learning machines for first impressions analysis. We describe the setting, scenario, evaluation metrics and preliminary outcomes of the competition. To the best of our knowledge this is the first effort in terms of challenges for explainability in computer vision. In addition our challenge design comprises several other quantitative and qualitative elements of novelty, including a “coopetition” setting, which combines competition and collaboration

show abstract

“…Amazon Mechanical Turk (AMT) was used for collecting the data to generate these labels. In order to guarantee the reliability of the labels, all rankings provided by the AMT workers were converted into normalized real valued scores using a principled procedure (see [1] for details). Personality traits that were used in the dataset were those from the Five Factor Model (also known as the Big Five).…”

Section: The First Impressions Data Setmentioning

confidence: 99%

Visualizing Apparent Personality Analysis with Deep Residual Networks

Güçlütürk

Güçlü

Pérez

et al. 2017

2017 IEEE International Conference on Computer Vision Workshops (ICCVW)

Self Cite

View full text Add to dashboard Cite

Automatic prediction of personality traits is a subjective task that has recently received much attention. Specifically, automatic apparent personality trait prediction from multimodal data has emerged as a hot topic within the filed of computer vision and, more particularly, the so called "looking at people" sub-field. Considering "apparent" personality traits as opposed to real ones considerably reduces the subjectivity of the task. The real world applications are encountered in a wide range of domains, including entertainment, health, human computer interaction, recruitment and security. Predictive models of personality traits are useful for individuals in many scenarios (e.g., preparing for job interviews, preparing for public speaking). However, these predictions in and of themselves might be deemed to be untrustworthy without human understandable supportive evidence. Through a series of experiments on a recently released benchmark dataset for automatic apparent personality trait prediction, this paper characterizes the audio and visual information that is used by a state-of-the-art model while making its predictions, so as to provide such supportive evidence by explaining predictions made. Additionally, the paper describes a new web application, which gives feedback on apparent personality traits of its users by combining model predictions with their explanations.

show abstract

Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Cited by 38 publications

References 26 publications

Modeling, Recognizing, and Explaining Apparent Personality From Videos

Modeling, Recognizing, and Explaining Apparent Personality From Videos

Design of an explainable machine learning challenge for video interviews

Visualizing Apparent Personality Analysis with Deep Residual Networks

Contact Info

Product

Resources

About