Assessing the importance of audio/video synchronization for simultaneous translation of video sequences

Abstract

Lip synchronization is considered a key parameter during interactive communication. In the case of video conferencing and television broadcasting, the differential delay between audio and video should remain below certain thresholds, as recommended by several standardization bodies. However, further research has also shown that these thresholds can be relaxed, depending on the targeted application and use case. In this article, we investigate the influence of lip sync on the ability to perform real-time language interpretation during video conferencing. Furthermore, we are also interested in determining proper lip sync visibility thresholds applicable to this use case. Therefore, we conducted a subjective experiment using expert interpreters, which were required to perform a simultaneous translation, and using non-experts. Our results show that significant differences are obtained when conducting subjective experiments with expert interpreters. As interpreters are primarily focused on performing the si-N. Staelens · N. Vercammen · B. Vermeulen · P. Demeester

Subjective video quality evaluation of different content types under different impairments

“…As authors in Gaston, Boley, Selter, and Ratterman (2010) state, audio artifacts can either enhance or reduce the ability to detect a certain video impairment present in the same sequence. For some types of contents, synchronization between audio and video can be a key factor in the subjective assessment (Staelens, De Meulenaere, et al, 2012). Authors in Kim, Kondoz, and Shi (2013) used the correlation between audio and video as a new metric to predict the QoE, corroborating that audio and video mismatch affects the perceived audio quality.…”

Audio and Video Mixing Method to Enhance WebRTC

“…Similarly, we want the first audio frame of the mixed stream to be composited by the second audio frame of p 2 and the sixth frame of p 1 , but this is impossible. However, according to the ITU-R BT.1359 standard [41]- [43], when the difference T ranges from [−100 ms, 25 ms], it will be acceptable to the human eye. For this reason, we can redefine f O v (k).…”

“…In general, combining the ITU-R BT.1359 standard [41]- [43] can simplify Formula (3) into Formula (7), which can be used to quantitatively evaluate the synchronization of audio and video, where T represents the time difference on a certain standard time; gt a and gt v represent a certain standard time when audio and video were generated and, theoretically, gt a should be equal to gt v ; pt v indicates the actual time when the user sees the picture at that time point during live broadcast; and pt a indicates the time when the user hears the audio at that time during the live broadcast. The larger the T value is, the more out of sync the broadcast is.…”

Monitoring of audio visual quality by key indicators

“…In video conferencing, streaming and television broadcasting, the uneven delay between audio and video should remain below certain thresholds, recommended by several standardization bodies. However, research shows that the thresholds can be relaxed, depending on the targeted application and use case [21].…”

“…Regarding detection thresholds, [21] describes the high number of thresholds determined by the authors. Some authors and research groups have concluded that audio may be played up to 305 ms ahead of video and conversely video can be displayed up to 190 ms ahead of the audio.…”