Figure 1: Interaction opportunities of our design space: (A) Reflections allow users to interact with artifacts inside a museum cabinet. (B) Reflections can reveal internal details of real objects. (C) A Digital Musical Instrument augmented with projection mapping and a volumetric display. (D) This augmentation is visible from any point of view and from both sides of the mirror.

No abstract

As Virtual Reality headsets become accessible, more and more artistic applications are developed, including immersive musical instruments. 3D interaction techniques designed in the 3D User Interfaces research community, such as navigation, selection and manipulation techniques, open numerous opportunities for musical control. For example, navigation techniques such as teleportation, free walking/flying and path-planning enable different ways of accessing musical scores, scenes of spatialized sounds sources or even parameter spaces. Manipulation techniques provide novel gestures and metaphors, e.g. for drawing or sculpting sound entities. Finally, 3D selection techniques facilitate the interaction with complex visual structures which can represent hierarchical temporal structures, audio graphs, scores or parameter spaces. However, existing devices and techniques were developed mainly with a focus on efficiency, i.e. minimising error rate and task completion times. They were therefore not designed with the specifics of musical interaction in mind. In this paper, we review existing 3D interaction techniques and examine how they can be used for musical control, including the possibilities they open for instrument designers. We then propose a number of research directions to adapt and extend 3DUIs for musical expression KEYWORDS 3D user interfaces, Immersive virtual musical instruments, Virtual Reality, New interface for musical expression