The Effects of Background Noise on User Experience and Performance of Mixed Reality Voice Dictation

Abstract

From education to manufacturing and medicine, the Microsoft HoloLens 2 mixed reality headset has been increasingly adopted. Text entry is a core functionality of the HoloLens 2, enabling communication and search features. The device tracks hand and finger movements interacting with a virtual keyboard but is slow and cumbersome. Speech-to-text dictation is a promising alternative requiring less physical effort and time. However, the HoloLens 2 dictation performance in environmental settings with varying noise levels is unclear. This study investigated the text input accuracy (WER), speed (WPM), and user perceptions of the HoloLens 2 dictation across three background noise levels (40 dB, 55 dB, and 70 dB). The 70 dB condition resulted in inferior outcomes for text input speed, perceived usability, mental demand, temporal demand, and effort. Users also reported lengthy processing delays in the 70 dB condition, while accuracy remained consistent (2–5% WER) across all noise conditions.

Exploring the user experience (UX) of a multi-window augmented reality environment

“…The 50-min use of the AR headset in the study led to concerning levels of overall simulator sickness symptoms (M = 18.51, SD = 16.48), with significant levels of nausea (M = 11.93, SD = 12.71) as well as disorientation (M = 23.66, SD = 36.16) and oculomotor discomfort (M = 27.29, SD = 22.59) being classified as bad (Stanney et al, 1997). Despite employing different tasks, this is in line with a previous study (Sinlapanuntakul P. et al, 2022), which found similarly concerning levels of simulator sickness symptoms (M = 18.14, SD = 13.69) among participants (N = 20) using the Microsoft HoloLens 2 for 40 min. These results suggest that the duration of an AR device use may be a major contributor to the development of simulator sickness, even without verbal feedback of frustrations or discomfort caused by it (Stanney et al, 1997).…”

Impact of Industrial Noise on Speech Interaction Performance and User Acceptance when Using the MS HoloLens 2

“…To mitigate harmful effects and reflect industrial conditions, experiments were carried out wearing hearing protection, enabling realism. In summary, previous results show that the HL1 has a WER of 55% (at 70 dB(A); see [18]) and the HL2 according to [19] has an average WER of 51% (SPL 40-70 dB(A)). In this study, a WER of 37% was demonstrated at 87 dB(A) with a German text template.…”

“…Analogously, comparing the performance of the gesture control on the HL1, the WER was about 0.12%; see [10]. In contrast with [19], a significant influence of the SPL on the WER was shown. Although [19] did not find a significant effect on the perceived system usability, this contribution did confirm a significant difference in performance and effort expectancy (see UTAU2 PE and EE) by manipulating the experimental orders with start error and end error conditions.…”

“…The mean WER was measured at various SPL conditions of 36%, 68% and 48%. No statistically significant differences were found for the perceived system usability in the given experimental setup [19].…”

“…In accordance with [18], no statistically significant differences could be demonstrated between the SPL effects on the HL2 using the WER. In contrast, the analogous research design on the HL1 (see [19]) revealed a significant effect (F(1.37, 20.47) = 13.56, p < 0.05, η 2 p = 0.475). In order to estimate an appropriate sample size and to demonstrate the concept of the research design, a pre-study with 5 participants was conducted, from which the first data were generated and calculated; see Table 2.…”