Advances in Online Audio-Visual Meeting Transcription

Yoshioka, Takuya; Abramovski, Igor; Aksoylar, Cem; Chen, Zhuo; David, Moshe; Dimitriadis, Dimitrios; Gong, Yifan; Gurvich, Ilya; Huang, Xuedong; Huang, Yan; Hurvitz, Aviv; Jiang, Li; Koubi, Sharon; Krupka, Eyal; Leichter, Ido; Liu, Changliang; Parthasarathy, Partha; Vinnikov, Alon; Wu, Lingfeng; Xiao, Xiong; Xiong, Wei; Wang, Huaming; Wang, Zhenghao; Zhang, Jun; Zhao, Yong; Zhou, Tianyan

doi:10.48550/arxiv.1912.04979

Cited by 9 publications

(4 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, utilizing a multi-modal approach by adding video data has been shown to outperform audio-only analysis in the field of speech processing [21]. Thus, recent studies have often leveraged a multi-stage approach and incorporated video data, rather than using a single end-to-end model [22][23][24][25][26][27]. The method proposed by Yoshioka et al [22] uses face tracking and identification, sound source localization, and speaker identification, yet it requires multi-channel audio input.…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

“…Thus, recent studies have often leveraged a multi-stage approach and incorporated video data, rather than using a single end-to-end model [22][23][24][25][26][27]. The method proposed by Yoshioka et al [22] uses face tracking and identification, sound source localization, and speaker identification, yet it requires multi-channel audio input. Another method, initially introduced by Nagrani et al [23], first performs face detection and tracking, and then uses active speaker detection (ASD) to determine the synchronization between the mouth movement and speech in the video to identify the speaker.…”

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

See 1 more Smart Citation

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Zeulner,

Hagerer,

Müller

et al. 2024

Information

View full text Add to dashboard Cite

Current methods for assessing individual well-being in team collaboration at the workplace often rely on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal was to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We focus on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to explore which features are predictors of self-reported satisfaction scores. We applied the toolchain to a case study, where we collected videos of 20 teams with 56 participants collaborating over a four-day period in a team project in an educational environment. Our audiovisual speaker diarization extracted individual speech features from a noisy environment. As the dependent variable, team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores were predicted using speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain was able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

show abstract

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

Section: Multi-modal Speaker Diarizationmentioning

confidence: 99%

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Zeulner,

Hagerer,

Müller

et al. 2024

Information

View full text Add to dashboard Cite

show abstract

“…Many recent source separation systems assume that the number of active sources in a mixture is known in advance during both training and inference phases [1][2][3][4][5][6][7][8][9][10][11][12][13][14]. Such assumption can be valid when there is additional information, such as visual cue or source locations [15,16], however for a general blind source separation system it is typically not straightforward to obtain such information, especially in inference phase. In problems such as the separation of shorter streams or chunks in a long mixture, e.g.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we propose a simple training method based on the fixed-output assumption by designing proper training targets for the invalid outputs. We adopt the fixed-output-number assumption as in real-world conversations such as meeting scenarios, the maximum number of simultaneously active speakers is almost always fewer than three [15,24], thus a maximum number of speakers can typically be pre-assumed. Instead of using low-energy auxiliary targets for invalid outputs, we use the mixture itself as auxiliary targets to force the invalid outputs to perform autoencoding.…”

Section: Introductionmentioning

confidence: 99%

Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss

Luo

Mesgarani

2020

Interspeech 2020

View full text Add to dashboard Cite

Many recent source separation systems are designed to separate a fixed number of sources out of a mixture. In the cases where the source activation patterns are unknown, such systems have to either adjust the number of outputs or to identify invalid outputs from the valid ones. Iterative separation methods have gain much attention in the community as they can flexibly decide the number of outputs, however (1) they typically rely on long-term information to determine the stopping time for the iterations, which makes them hard to operate in a causal setting;(2) they lack a "fault tolerance" mechanism when the estimated number of sources is different from the actual number. In this paper, we propose a simple training method, the auxiliary autoencoding permutation invariant training (A2PIT), to alleviate the two issues. A2PIT assumes a fixed number of outputs and uses auxiliary autoencoding losses to force the invalid outputs to be the copies of the input mixture, and detects invalid outputs in a fully unsupervised way during inference phase. Experiment results show that A2PIT is able to improve the separation performance across various numbers of speakers and effectively detect the number of speakers in a mixture.

show abstract

Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss

Luo

Mesgarani

2020

Interspeech 2020

View full text Add to dashboard Cite

Advances in Online Audio-Visual Meeting Transcription

Cited by 9 publications

References 50 publications

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss

Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss

Contact Info

Product

Resources

About