Watch, Listen Once, and Sync: Audio-Visual Synchronization With Multi-Modal Regression Cnn

Kikuchi, Tetsuhiro; Ozasa, Yuko

doi:10.1109/icassp.2018.8461853

Cited by 6 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve these shortcomings, researchers have proposed some improvement algorithms. For example, the works in [12,23] utilized the correlation between audio-visual signals for synchronization detection. They extract lip pictures in video frames and then compare them with the features of audio signal through a deep-learning-based model to determine the synchronization status of audio-visual signals.…”

Section: Related Workmentioning

confidence: 99%

Timestamp-independent Haptic-Visual Synchronization

Xu¹,

Huang²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

The booming haptic data significantly improves the users' immersion during multimedia interaction. As a result, the study of Haptic, Audio-Visual Environment (HAVE) has attracted attentions of multimedia community. To realize such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to user experience. Despite of audio-visual synchronization efforts, there is still a lack of haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic-visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a hapticvisual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics, and a key frame extraction of visual signals based on deep object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic-visual signals. Without timestamps in signal flow, the proposed method is still effective and more robust to complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

Timestamp-independent Haptic-Visual Synchronization

Xu¹,

Huang²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To solve these shortcomings, researchers have proposed some improvement algorithms. For example, the works in [ 19 , 20 ] utilized the correlation between audio-visual signals for synchronization detection. They extract lip pictures in video frames and then compare them with the features of an audio signal through a deep-learning-based model to determine the synchronization status of audio-visual signals.…”

Section: Introductionmentioning

confidence: 99%

A Timestamp-Independent Haptic–Visual Synchronization Method for Haptic-Based Interaction System

Huang

Zhao

et al. 2022

Sensors

View full text Add to dashboard Cite

The booming haptic data significantly improve the users’ immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic–visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic–visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic–visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.

show abstract