This paper investigates how the perceived physicality of the action of applying force with a finger on a rigid surface (such as on a force-sensing touch screen) can be enhanced using real-time synthesized audio feedback. A selection of rich and evocative audio designs was used. Additionally, audio-tactile cross-modal integration was encouraged, by observing that the main rules of multisensory integration were supported. The study conducted showed that richness of perceived physicality increased considerably, mostly in its auditory expression (what pressing sounded like). In addition, in many instances it was observed that the haptic expression of physicality also increased (what pressing felt like), including some perception of compliance. This last result was particularly interesting as it showed that audio-tactile cross-modal integration might be present.

In this paper we describe AHNE (Audio-Haptic Navigation Environment). It is a three-dimensional user interface (3D UI) for manipulating virtual sound objects with natural gestures in a real environment. AHNE uses real-time motion tracking and custom-made glove controllers as input devices, and auditory and haptic feedback as the output. We present the underlying system and a possible use for the interface as a musical controller.