Who said that?: Audio-visual speaker diarisation of real-world meetings

Chung, Joon Son; Lee, Bong-Jin; Han, Icksang

doi:10.48550/arxiv.1906.10042

Cited by 4 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, utilizing a multi-modal approach by adding video data has been shown to outperform audio-only analysis in the field of speech processing [21]. Thus, recent studies have often leveraged a multi-stage approach and incorporated video data, rather than using a single end-to-end model [22][23][24][25][26][27]. The method proposed by Yoshioka et al [22] uses face tracking and identification, sound source localization, and speaker identification, yet it requires multi-channel audio input.…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

“…After that, the approach uses face verification to classify whether the recognized face belongs to a specific individual. However, this requires prior knowledge such as the number and images of the speakers, see also Chung et al [24,27]. To solve this problem, this step was replaced by the clustering of face tracks in the latest version [25].…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

See 1 more Smart Citation

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Zeulner,

Hagerer,

Müller

et al. 2024

Information

View full text Add to dashboard Cite

Current methods for assessing individual well-being in team collaboration at the workplace often rely on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal was to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We focus on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to explore which features are predictors of self-reported satisfaction scores. We applied the toolchain to a case study, where we collected videos of 20 teams with 56 participants collaborating over a four-day period in a team project in an educational environment. Our audiovisual speaker diarization extracted individual speech features from a noisy environment. As the dependent variable, team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores were predicted using speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain was able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

show abstract

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Zeulner,

Hagerer,

Müller

et al. 2024

Information

View full text Add to dashboard Cite

show abstract

“…Thus, the most reliable and least intruding way of data collection uses one 360°room camera and a room microphone. Recent studies often leveraged a multi-stage approach when incorporating video data, rather than using a single end-to-end model [12][13][14][15][16][17]. The method proposed by Yoshioka et al [12] uses face tracking and identification, sound source localization, and speaker identification, yet it requires multi-channel audio input.…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

“…After that, the approach uses face verification to classify whether the recognized face belongs to a specific individual. However, this requires prior knowledge such as the number and images of the speakers, see also Chung et al [14,17]. To solve this problem, this step was replaced by the clustering of face tracks in the latest version [15].…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

Predicting Individual Well-Being in Teamwork Contexts Based on Multi-Modal Speech Features

Zeulner,

Hagerer,

Mueller

et al. 2024

Preprint

View full text Add to dashboard Cite

Current methods for assessing individual well-being in team collaboration at the workplace rely often on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal is to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We are focusing on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to find which features are predictors of self-reported satisfaction scores. We apply the toolchain to a case study where we collected videos of 22 teams with 56 participants collaborating over a four-day period in a team project. Our audiovisual speaker diarization extracts individual speech features in a noisy environment. As the dependent variable team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores have been predicted with speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain is able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

show abstract

“…McDorman states that the context of what is verbally discussed in a meeting is important [22]. Previous work on verbal information analysis in meetings often focuses on creating transcriptions of the meetings [23], [24], [25].…”

Section: A Meeting Behavior Analysismentioning

confidence: 99%

EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network

Watanabe,

Sathyanarayana,

Dengel

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Measuring the level of engagement among participants in a meeting is crucial for evaluating collective understanding. While previous studies have utilized multiple sensors, such as wearable devices, to gauge engagement levels in offline environments, the shift to remote meetings due to the COVID-19 pandemic presents new challenges. In this study, we propose a method for measuring student engagement during online meetings using only the built-in web cameras on their devices. We collect high, middle, and low engagement level recording data from 24 students. We decided to collect data using the role-acting approach instead of conventional self-reporting or post-experiment annotation. With the feature extraction based approach, we achieved a classification rate of 46.7%. With a deep learning based approach, we achieved a classification rate of 89.5% using MobileNetV2 for leave-one-participant-out cross-validation, which demonstrated higher accuracy than previous studies. From the model, we implement an application EnGauge and conduct a pilot study. The results demonstrate a new approach to data collection, an optimal engagement level recognition model, and application scenarios.

show abstract

Who said that?: Audio-visual speaker diarisation of real-world meetings

Cited by 4 publications

References 29 publications

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Predicting Individual Well-Being in Teamwork Contexts Based on Multi-Modal Speech Features

EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network

Contact Info

Product

Resources

About