Augmented Reality (AR) and Mixed Reality (MR) technology has potential for supporting mobile applications. However, nonvisual interaction modalities are undervalued and underused in AR/MR applications. Visual displays can be ineffective or inappropriate in some situations such as walking or driving. Meanwhile, nonvisual modalities are becoming increasingly important in mobile user experiences. In this paper, we report two studies investigating nonvisual interaction modalities such as audio and haptic displays with mobile AR/MR applications. In the first study, we investigate a range of design factors for haptic and audio displays, including rhythm, amplitude, and their combination in representing tourism information to users with a mobile phone. The results show a main effect for Interaction modality, with identification rates highest for information represented in a combined Haptic-Audio display. In the second study, we investigate target location tasks in 3D space using spatial audio feedback and a head-mounted display. We evaluate several design factors including audio feedback device, volume, rhythm, and the target’s horizontal and vertical position. Results show that the vertical positions are very difficult to locate, and overall our participants prefer audio cues with loud volume and fast rhythm. Finally, we propose practical audio and haptic display design guidelines for AR/MR applications.

Augmented Reality (AR) technology has potential for supporting applications such as tourism. However, non-visual interaction modalities are undervalued and underused in AR tourism applications. Visual displays are ineffective or inappropriate in some situations such as in strong sunlight or when walking or driving.Meanwhile, non-visual modalities are becoming increasingly important in mobile user experiences. In this paper, two non-visual interaction modalities, haptic display and audio display, and their combination are evaluated in representing tourism information to users with a mobile phone.An experimental evaluation was conducted with different tourism information presented by haptic display, audio display and both, with 3 different rhythms and 3 levels of amplitude. The results show a main effect of interaction modality, with identification rate highest for information represented in the combined Haptic-Audio display at 86.7%, while no significant effect was found for rhythm or amplitude alone. Qualitative data from the participants indicated that, across all interaction modalities, different levels of amplitude were more difficult to distinguish than different rhythms or different combinations of rhythm and amplitude.