Generating Personalized Behavioral Feedback for a Virtual Job Interview Training System Through Adversarial Learning

Heimerl, Alexander; Mertes, Silvan; Schneeberger, Tanja; Baur, Tobias; Liu, Ailin; Becker, Linda; Rohleder, Nicolas; Gebhard, Patrick; André, Elisabeth

doi:10.1007/978-3-031-11644-5_67

Cited by 4 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Virtual job interview-trainings can increase performance during the interview, and may therefore increase hiring chances (Smith et al, 2021 ). A first prototype which is based on our job interview-scenario has already been developed in our group (Heimerl et al, 2022b ). In future studies, this system will be further developed and evaluated (e.g., with respect to perceived stress and biological stress responses).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario

2023

Self Cite

View full text Add to dashboard Cite

IntroductionSince the COVID-19 pandemic, working environments and private lives have changed dramatically. Digital technologies and media have become more and more important and have found their way into nearly all private and work environments. Communication situations have been largely relocated to virtual spaces. One of these scenarios is digital job interviews. Job interviews are usually—also in the non-digital world—perceived as stressful and associated with biological stress responses. We here present and evaluate a newly developed laboratory stressor that is based on a digital job interview-scenario.MethodsN = 45 healthy people participated in the study (64.4% female; mean age: 23.2 ± 3.6 years; mean body mass index = 22.8 ± 4.0 kg/m2). Salivary alpha-amylase (sAA) and cortisol were assessed as measures for biological stress responses. Furthermore, perceived stress was rated at the time points of the saliva samplings. The job interviews lasted between 20 and 25 min. All materials, including instructions for the experimenter (i.e., the job interviewer) and the data set used for statistical analysis, as well as a multimodal data set, which includes further measures, are publicly available.ResultsTypical subjective and biological stress-response patterns were found, with peak sAA and perceived stress levels observed immediately after the job interviews and peak cortisol concentrations 5 min afterwards. Female participants experienced the scenario as more stressful than male participants. Cortisol peaks were higher for participants who experienced the situation as a threat in comparison to participants who experienced it as a challenge. Associations between the strength of the stress response with further person characteristics and psychological variables such as BMI, age, coping styles, and personality were not found.DiscussionOverall, our method is well-suited to induce biological and perceived stress, mostly independent of person characteristics and psychological variables. The setting is naturalistic and easily implementable in standardized laboratory settings.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Furthermore, they were equipped with a finger-clip sensor (iom) on the non-dominant hand. This data is reported in Heimerl et al (2023) as part of the annotated multimodal data set.…”

Section: Further Recordingsmentioning

confidence: 99%

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario

2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similarly, if any two consecutive beats occur less than 333.33 milliseconds apart (heart rate > 180 beats per minute), then we consider the segment to have points that are wrongly detected as beats. Using NeuroKit2 (Makowski et al, 2021) python library, we compute 22 HRV features from the time domain, frequency domain, and poincaré plots Heimerl et al, 2023).…”

Section: Hrv Featuresmentioning

confidence: 99%

Flow in human-robot collaboration—multimodal analysis and perceived challenge detection in industrial scenarios

Prajod,

Lavit Nicora,

Mondellini

et al. 2024

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Introduction: Flow state, the optimal experience resulting from the equilibrium between perceived challenge and skill level, has been extensively studied in various domains. However, its occurrence in industrial settings has remained relatively unexplored. Notably, the literature predominantly focuses on Flow within mentally demanding tasks, which differ significantly from industrial tasks. Consequently, our understanding of emotional and physiological responses to varying challenge levels, specifically in the context of industry-like tasks, remains limited.Methods: To bridge this gap, we investigate how facial emotion estimation (valence, arousal) and Heart Rate Variability (HRV) features vary with the perceived challenge levels during industrial assembly tasks. Our study involves an assembly scenario that simulates an industrial human-robot collaboration task with three distinct challenge levels. As part of our study, we collected video, electrocardiogram (ECG), and NASA-TLX questionnaire data from 37 participants.Results: Our results demonstrate a significant difference in mean arousal and heart rate between the low-challenge (Boredom) condition and the other conditions. We also found a noticeable trend-level difference in mean heart rate between the adaptive (Flow) and high-challenge (Anxiety) conditions. Similar differences were also observed in a few other temporal HRV features like Mean NN and Triangular index. Considering the characteristics of typical industrial assembly tasks, we aim to facilitate Flow by detecting and balancing the perceived challenge levels. Leveraging our analysis results, we developed an HRV-based machine learning model for discerning perceived challenge levels, distinguishing between low and higher-challenge conditions.Discussion: This work deepens our understanding of emotional and physiological responses to perceived challenge levels in industrial contexts and provides valuable insights for the design of adaptive work environments.

show abstract

“…Studies have also developed algorithm‐based systems automatically analyzing behavioral parameters to train interviewees (e.g., Gebhard et al, 2019; Heimerl et al, 2022; Hoque et al, 2013; Langer et al, 2016; Lin‐Stephens et al, 2022). Langer et al (2016) automatically recorded interviewees' nonverbal behavior, which was then fed back to them in terms of behavioral recommendations.…”

Section: Machine Learning Analyses Of Applicant Behavior In Interviewsmentioning

confidence: 99%

“…The intervention helped interviewees reduce interview anxiety and improved performance and perceived hireability. Heimerl et al (2022) used a combination of automatically extracted facial and body behavioral features and conversational cues (turn‐taking computed from vocal cues) to generate textual feedback via a generative adversarial network (GAN)‐based approach. Few studies have focused on feedback related to how applicants should modulate what they say, that is, their verbal behavior, even though the effect of verbal behavior on employment decisions is at least as important as those of nonverbal or paraverbal behavior (Burnett & Motowidlo, 1998; Hollandsworth et al, 1979; Rasmussen, 1984).…”

Section: Machine Learning Analyses Of Applicant Behavior In Interviewsmentioning

confidence: 99%

Automatic identification of storytelling responses to past‐behavior interview questions via machine learning

Bangerter

Mayor

Muralidhar

et al. 2023

Int J Selection Assessment

View full text Add to dashboard Cite

Structured interviews often feature past‐behavior questions, where applicants are asked to tell a story about past work experience. Applicants often experience difficulties producing such stories. Automatic analyses of applicant behavior in responding to past‐behavior questions may constitute a basis for delivering feedback and thus helping them improve their performance. We used machine learning algorithms to predict storytelling in transcribed speech of participants responding to past‐behavior questions in a simulated selection interview. Responses were coded as to whether they featured a story or not. For each story, utterances were also manually coded as to whether they described the situation, the task/action performed, or results obtained. The algorithms predicted whether a response features a story or not (best accuracy: 78%), as well as the count of situation, task/action, and response utterances. These findings contribute to better automatic identification of verbal responses to past‐behavior questions and may support automatic provision of feedback to applicants about their interview performance.

show abstract

Generating Personalized Behavioral Feedback for a Virtual Job Interview Training System Through Adversarial Learning

Cited by 4 publications

References 13 publications

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario

Flow in human-robot collaboration—multimodal analysis and perceived challenge detection in industrial scenarios

Automatic identification of storytelling responses to past‐behavior interview questions via machine learning

Contact Info

Product

Resources

About