Influence of visual cues on head and eye movements during listening tasks in multi-talker audiovisual environments with animated characters

Abstract

Recent studies of hearing aid benefits indicate that head movement behavior influences performance. To systematically assess these effects, movement behavior must be measured in realistic communication conditions. For this, the use of virtual audiovisual environments with animated characters as visual stimuli has been proposed. It is unclear, however, how these animations influence the head-and eye-movement behavior of subjects. Here, two listening tasks were carried out with a group of 14 young normal hearing subjects to investigate the influence of visual cues on head-and eye-movement behavior; on combined localization and speech intelligibility task performance; as well as on perceived speech intelligibility, perceived listening effort and the general impression of the audiovisual environments. Animated characters with different lip-syncing and gaze patterns were compared to an audio-only condition and to a video of real persons. Results show that movement behavior, task performance, and perception were all influenced by visual cues. The movement behavior of young normal hearing listeners in animation conditions with lip-syncing was similar to that in the video condition. These results in young normal hearing listeners are a first step towards using the animated characters to assess the influence of head movement behavior on hearing aid performance.

The Virtual Reality Lab: Realization and Application of Virtual Sound Environments

“…To be able to control the realism of head movement, the experimental factors influencing the movement need to be identified. In that regard, Hendrikse et al (2018) found that subjects moved their heads and eyes much less in turn-taking conversations if no visual stimulation was provided. Furthermore, Hendrikse et al (2018) showed that subjects changed their movement behavior dependent on the head and gaze behavior of animated characters used as simulated conversation partners.…”

“…In that regard, Hendrikse et al (2018) found that subjects moved their heads and eyes much less in turn-taking conversations if no visual stimulation was provided. Furthermore, Hendrikse et al (2018) showed that subjects changed their movement behavior dependent on the head and gaze behavior of animated characters used as simulated conversation partners. This shows that measures of head and eye movements may be used as a metric to assess experimental paradigms and their level of ecological validity*.…”

“… *Note that head and eye movements were used both as a metric and as experimental factors by Hendrikse et al (2018) : head and eye movements of communication partners in the (simulated) experiment were used as an experimental factor that was found to influence the metric, that is, the head and eye movement of the subjects in the experiment. †The sound events and levels experienced by each subject during the walks were not controlled.…”

“…Whereas the quality of acoustic and visual stimulation (items 1 and 2) can be controlled parametrically using current virtual reality technology, and, as mentioned earlier, several studies relating the factors of acoustic and visual stimulation with the corresponding metrics have been published (e.g., Hendrikse et al 2018 , 2019 , 2020 ), items 3 and 4 are largely under-investigated, to the knowledge of the authors. On the basis of the notion of the relevance of bodily acts in communication ( Mead 1956 , p. 98), we may assume that head movements, as well as other gestures, such as leaning forward or backward, or arm and hand movements, are important observables/metrics for estimating the level of involvement, and thus the level of ecological validity, also for items 3 and 4.…”

“…This section gives a brief summary of the current state of development of the virtual reality lab at Oldenburg University. For a more comprehensive overview of the technology, see Llorach et al (2018) , and for a detailed description of the setup, see Hendrikse et al (2018 , 2019 ).…”