Nonverbal Social Sensing in Action: Unobtrusive Recording and Extracting of Nonverbal Behavior in Social Interactions Illustrated with a Research Example

Abstract

Nonverbal behavior coding is typically conducted by ''hand''. To remedy this time and resource intensive undertaking, we illustrate how nonverbal social sensing, defined as the automated recording and extracting of nonverbal behavior via ubiquitous social sensing platforms, can be achieved. More precisely, we show how and what kind of nonverbal cues can be extracted and to what extent automated extracted nonverbal cues can be validly obtained with an illustrative research example. In a job interview, the applicant's vocal and visual nonverbal immediacy behavior was automatically sensed and extracted. Results show that the applicant's nonverbal behavior can be validly extracted. Moreover, both visual and vocal applicant nonverbal behavior predict recruiter hiring decision, which is in line with previous findings on manually coded applicant nonverbal behavior. Finally, applicant average turn duration, tempo variation, and gazing best predict recruiter hiring decision. Results and implications of such a nonverbal social sensing for future research are discussed.

Examining digital interviews for personnel selection: Applicant reactions and interviewer ratings

“…In contrast to videoconference interviews, digital interviews provide fewer communication channels and thus less social bandwidth. For instance, interviewers and interviewees do not see each other, they cannot direct nonverbal behavior at each other, and they cannot use backchanneling behavior (e.g., nodding; Frauendorfer, Schmid Mast, Nguyen, & Gatica‐Perez, ).…”

Triadic Synchrony: Application of Multiple Wavelet Coherence to a Small Group Conversation

“…In the early stage of nonverbal research, generating time-series movement data or coding behaviors has been resource intensive, or even painstaking, which hindered the theoretical and/or practical development of (nonverbal) behavioral research on a group interaction. Some recent studies began to utilize automated techniques to generate time-series movement data [11]- [15]. These novel techniques could reduce the cost of conducting nonverbal research, which will enable us to obtain a better understanding of small group interaction from the perspective of body movement and nonverbal behavior.…”

“…Coding behaviors is time-consuming and painstaking in itself and requires establishing reliability among coders. To address this problem, some recent studies have utilized automated techniques to generate time-series movement data; they use a depth sensor, Kinect (Microsoft) [11], or employ video-tracking techniques [12]- [15]. Behavioral data acquired by these techniques would be less costly as well as highly reliable.…”

Disfluent Responses to Job Interview Questions and What They Entail

“…We investigated these two research questions using data gathered for another research project on sensing of candidate nonverbal behavior (Frauendorfer, Mast, Nguyen, & Gatica-Perez, 2014). Candidates applied for a research assistant position.…”