Mining a multimodal corpus of doctor’s training for virtual patient’s feedbacks

Abstract

Doctors should be trained not only to perform medical or surgical acts but also to develop competences in communication for their interaction with patients. For instance, the way doctors deliver bad news has a significant impact on the therapeutic process. In order to facilitate the doctors' training to break bad news, we aim at developing a virtual patient ables to interact in a multimodal way with doctors announcing an undesirable event.One of the key elements to create an engaging interaction is the feedbacks' behavior of the virtual character. In order to model the virtual patient's feedbacks in the context of breaking bad news, we have analyzed a corpus of real doctor's training. The verbal and nonverbal signals of both the doctors and the patients have been annotated. In order to identify the types of feedbacks and the elements that may elicit a feedback, we have explored the corpus based on sequences mining methods. Rules, that have been extracted from the corpus, enable us to determine when a virtual patient should express which feedbacks when a doctor announces a bad new.

Modeling Feedback in Interaction With Conversational Agents—A Review

“…Second, to mark that the speaker can take the turn back and continue speaking, and third, to express an attitudinal reaction to the content presented by the speaker, often in combination with verbal feedback. Porhet et al (2017) found that head movements are the most common feedback modality both from doctors toward their patients and vice versa in a medical interaction corpus. The authors also found that a nod from the speaking doctor toward the listening patient is followed by a nod in response from the patient 29% of the time.…”

An integrated model for predicting backchannel feedbacks

“…Moreover, we also identified sequences of multimodal features (mainly POS tags and gestures) that could be used as backchannel predicting cues [14]. From these sequences, several rules have been extracted for generating backchannels:…”

Evaluating Temporal Predictive Features for Virtual Patients Feedbacks

“…Different gestures of both doctors and patients have been annotated: head movements, posture changes, gaze direction, eyebrow expressions, hand gestures, and smiles. More details on the corpus are presented in [13]. Three annotators -paid graduate students in linguistics -coded the corpus, with 5% double-checked for validation (Cohen's Kappa=0.63).…”