Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data

Torres, Michelle; Cantú, Francisco

doi:10.1017/pan.2021.9

Cited by 44 publications

(33 citation statements)

References 39 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results contribute to scholarship on images as data (Torres and Cantú 2022, Aslett et al 2020, Casas and Williams 2019, how political elites use social media (Russell 2021) to present their 'best face' to the public Carpinella et al (2016), Bauer, Kalmoe and, and how gender constrains the behavior of political leaders. At the same time, our work presents a new mechanism of understanding candidate strategy, social media, and the use of images in politics.…”

Section: Introductionsupporting

confidence: 56%

Mad, sad, but mostly glad: how men and women in politics communicate using emotions in images

Boussalis¹,

Coan²,

Holman³

2022

Preprint

View full text Add to dashboard Cite

Political leaders use emotions to attract support from voters, take positions on issues or policies, and enhance communication. Yet men and women are constrained by gender role expectations, which limit the range and type of acceptable emotions that a leader can express. In this paper, we examine how gender shapes the political communication of emotion through a new measure: emotional expression in images posted by leaders on social media. We examine more than 450,000 Facebook posts by Members of Congress (MOCs), extracting images from these posts, faces from those images, and identifying MOCs faces out to identify when MOCs post images of themselves as angry, happy, or expressing any emotion at all. Specifically, we develop a custom facial recognition model to identify MOCs in their social media imagery and use a convolutional neural network to detect emotional displays. Drawing on research on role congruity, we argue (and find) that women MOCs will post more images of themselves being happy and expressing any emotion and fewer images of themselves as angry. The project offers an opportunity to understand how political leaders use emotions and images to communicate to constituents and how gender shapes these strategic actions.

show abstract

Section: Introductionsupporting

confidence: 56%

Mad, sad, but mostly glad: how men and women in politics communicate using emotions in images

Boussalis¹,

Coan²,

Holman³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We build upon burgeoning scholarship that uses computational methods to study images as data (e.g., Cantú 2019;Casas and Williams 2019;Torres and Cantú 2021) and to capture and analyze facial expressions of political actors (e.g., Joo, Bucy, and Seidel 2019). While there is a strong interest in the nonverbal communication literature for increasingly granular measures of facial expressions, the field continues to be hampered by the methodological challenges involved with manually analyzing the content of images of faces at large scales-for instance, every frame of a set of hours-long debate videos.…”

Section: Emotional Expression Via Candidate Facial Displaysmentioning

confidence: 99%

Gender, Candidate Emotional Expression, and Voter Reactions During Televised Debates

et al. 2021

View full text Add to dashboard Cite

Voters evaluate politicians not just by what they say, but also how they say it, via facial displays of emotions and vocal pitch. Candidate characteristics can shape how leaders use—and how voters react to—nonverbal cues. Drawing on role congruity expectations, we study how the use of and reactions to facial, vocal, and textual communication in political debates varies by candidate gender. Relying on full-length videos of four German federal election debates (2005–2017) and a minor party debate, we use video, audio, and text data to measure candidate facial displays of emotion, vocal pitch, and speech sentiment. Consistent with our expectations, Angela Merkel expresses less anger than her male opponents, but she is just as emotive in other respects. Combining these measures of emotional expression with continuous responses recorded by live audiences, we find that voters punish Merkel for anger displays and reward her happiness and general emotional displays.

show abstract

“…Recently, the use of visual data also includes video data for analyzing social processes, such as police violence (Nassauer and Legewie, 2021). For larger visual data and in a quantitative design, scholars have to train algorithms "learning to see" objects of interest (Torres and Cantú, 2022). Nonetheless, although these contributions are critical for research designs relying on image data, there is a need for deeper grounding in the causal inference literature (Daoud and Dubhashi, 2023;Morgan and Winship, 2015;Imai, 2022;Pearl, 2009), creating a knowledge gap about how to leverage images for causal inference.…”

Section: Introductionmentioning

confidence: 99%

Integrating Earth Observation Data into Causal Inference: Challenges and Opportunities

Jerzak¹,

Johansson²,

Daoud³

2023

Preprint

View full text Add to dashboard Cite

Observational studies require adjustment for confounding factors that are correlated with both the treatment and outcome. In the setting where the observed variables are tabular quantities such as average income in a neighborhood, tools have been developed for addressing such confounding. However, in many parts of the developing world, features about local communities may be scarce. In this context, satellite imagery can play an important role, serving as a proxy for the confounding variables otherwise unobserved. In this paper, we study confounder adjustment in this non-tabular setting, where patterns or objects found in satellite images contribute to the confounder bias. Using the evaluation of anti-poverty aid programs in Africa as our running example, we formalize the challenge of performing causal adjustment with such unstructured data-what conditions are sufficient to identify causal effects, how to perform estimation, and how to quantify the ways in which certain aspects of the unstructured image object are most predictive of the treatment decision. Via simulation, we also explore the sensitivity of satellite image-based observational inference to image resolution and to misspecification of the image-associated confounder. Finally, we apply these tools in estimating the effect of anti-poverty interventions in African communities from satellite imagery.

show abstract

Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data

Cited by 44 publications

References 39 publications

Mad, sad, but mostly glad: how men and women in politics communicate using emotions in images

Mad, sad, but mostly glad: how men and women in politics communicate using emotions in images

Gender, Candidate Emotional Expression, and Voter Reactions During Televised Debates

Integrating Earth Observation Data into Causal Inference: Challenges and Opportunities

Contact Info

Product

Resources

About