Deciphering the Silent Participant

Abstract

Estimating a silent participant's degree of engagement and his role within a group discussion can be challenging, as there are no speech related cues available at the given time. Having this information available, however, can provide important insights into the dynamics of the group as a whole. In this paper, we study the classification of listeners into several categories (attentive listener, side participant and bystander). We devised a thin-sliced perception test where subjects were asked to assess listener roles and engagement levels in 15-second video-clips taken from a corpus of group interviews. Results show that humans are usually able to assess silent participant roles. Using the annotation to identify from a set of multimodal low-level features, such as past speaking activity, backchannels (both visual and verbal), as well as gaze patterns, we could identify the features which are able to distinguish between different listener categories. Moreover, the results show that many of the audiovisual effects observed on listeners in dyadic interactions, also hold for multi-party interactions. A preliminary classifier achieves an accuracy of 64%.

Towards Building an Attentive Artificial Listener: On the Perception of Attentiveness in Feedback Utterances

“…The fourth phase consisted of the students describing their projects' impact on society, and in the fifth and final phase they were asked to collaboratively come up with a joined project proposal. In a previous study, on the same corpus [13], it was found that third party observers could distinguish between at least three distinct types of listeners; an attentive listener, a side-participant, and a bystander. It was also found that the frequency of backchannel tokens was related to the perception of listener categories.…”

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

“…As Echeverria [10] provided a thorough survey of quantitative approach of analyzing group communications, recent studies exploited motion sensors and computer vision technologies to recognize human behaviors and automatically estimate the characteristics of the participants and meeting groups. With the goal of estimating the level of engagement of silent participants in multiparty communication, Oertel et al [28] proposed classifying group members into attentive listeners, side participants, and bystanders. They used verbal and nonverbal backchannels as prediction features.…”

Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

“…The recognition models using the features mentioned above were initially based on heuristic approaches [27,33,43]. Recent methods are based on machine learning techniques such as support vector machines (SVM) [18,23,36,42], hidden Markov models [30], and convolutional neural networks [41]. In this study, we focus on behaviors when the user is listening to system speech, such as backchannels, laughing, head nodding, and eye-gaze.…”

“…In this case, another approach is based on 'wisdom of crowds' where many annotators are recruited. Eventually, the annotations were integrated using methods such as majority labels, averaged scores, and agreed labels [18,22,23]. In our proposed method, the different views of the annotators are taken into account.…”

“…Furthermore, the investigation was extended to multi-party conversations. They analyzed features including audio and visual backchannels [23], eye-gaze behaviors [23,32], and upper body joints [42]. The recognition models using the features mentioned above were initially based on heuristic approaches [27,33,43].…”