Promise Into Practice: Application of Computer Vision in Empirical Research on Social Distancing

Abstract

Social scientists increasingly use video data, but large-scale analysis of its content is often constrained by scarce manual coding resources. Upscaling may be possible with the application of automated coding procedures, which are being developed in the field of computer vision. Here, we introduce computer vision to social scientists, review the state-of-the-art in relevant subfields, and provide a working example of how computer vision can be applied in empirical sociological work. Our application involves defining a ground truth by human coders, developing an algorithm for automated coding, testing the performance of the algorithm against the ground truth, and run the algorithm on a large-scale dataset of CCTV images. The working example concerns monitoring social distancing behavior in public space over more than a year of the COVID-19 pandemic. Finally, we discuss prospects for the use of computer vision in empirical social science research and address technical and ethical limitations.

Promise Into Practice: Application of Computer Vision in Empirical Research on Social Distancing

“…Comparison with cell phone-based activity level measures. At the time of writing, the computer vision algorithm is still being used to monitor outdoor activity levels and distances between people, to regularly inform the municipality of Amsterdam about compliance with social distancing directives in public space (Bernasco, Liebst, van der Veen et al 2021). They are also used in ongoing research addressing social distancing behavior during the COVID-19 pandemic, including a study on the effects of a night curfew (Liebst et al 2021a).…”

3D Social Research: Analysis of Social Interaction Using Computer Vision

“…The author shows that interparty interactions have decreased since the end of the 1990s, and that sparse inter-party interactions are predictive of party votes in later sessions. Bernasco et al (2022) 2022) study body language in political campaign videos in the U.S. They employ automatic pose detection and find that, in line with gender stereotypes, male candidates move their hands more assertively compared to female candidates.…”

“…The author shows that inter-party interactions have decreased since the end of the 1990s, and that sparse inter-party interactions are predictive of party votes in later sessions. Bernasco et al (2022) employ CV tools to analyze CCTV video data of public spaces during the Covid-19 pandemic. Compared to Hoeben et al's (2021) related study, which relies on human coding, Bernasco and colleagues analyze a much larger sample of videos to draw conclusions on social distancing violations, thus demonstrating the potential of CV tools for video data analysis of social interactions at scale.…”

“…However, to enable broader use of CV, researchers would benefit from a generalizable, flexible, and precise tool for measuring aspects such as physical distance, movement in space, and movement rate. For instance, both Dietrich and Sands (2021) and Bernasco et al (2022) went to great lengths to develop and validate strategies for reliably measuring physical distance between pedestrians.…”

“…Public distance between the two individuals was altogether absent during the interaction. By adjusting the distance threshold, we could use other distance categorizations, such as the social distancing guideline during the Covid-19 pandemic (Bernasco et al 2022; Hoeben et al 2021). Figure 4B shows the frequency of frames in which the two individuals were closer than 1.5 m, which constituted a violation of social distancing guidelines during the Covid-19 pandemic in many contexts.…”