A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots

Abstract

There is a significant amount of synergy between virtual reality (VR) and the field of robotics. However, it has only been in approximately the past five years that commercial immersive VR devices have been available to developers. This new availability has led to a rapid increase in research using VR devices in the field of robotics, especially in the development of VR interfaces for operating robots. In this paper, we present a systematic review on VR interfaces for robot operation that utilize commercially available immersive VR devices. A total of 41 papers published between 2016–2020 were collected for review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Papers are discussed and categorized into five categories: (1) Visualization, which focuses on displaying data or information to operators; (2) Robot Control and Planning, which focuses on connecting human input or movement to robot movement; (3) Interaction, which focuses on the development of new interaction techniques and/or identifying best interaction practices; (4) Usability, which focuses on user experiences of VR interfaces; and (5) Infrastructure, which focuses on system architectures or software to support connecting VR and robots for interface development. Additionally, we provide future directions to continue development in VR interfaces for operating robots.

VR as a 3D Modelling Tool in Engineering Design Applications

“…By immersing individuals in a virtual world where they may observe and interact with robots in 3D collaborative or distant locations, VR allows for more natural and intuitive interactions. This can improve situational awareness and make interaction simpler [22]. Alongside the improved spatial perception of the objects, virtual environments can enhance the communication between the dislocated participants as they enable speech and gesturing while at the same time viewing the product in true scale 3D environment when compared to the regular video conferencing tools [23].…”

Motion Polytopes in Virtual Reality for Shared Control in Remote Manipulation Applications

“…The last component of our framework integral to remote teleoperation is the interface, for which we utilize the medium of VR. In remote robot teleoperation, VR has become more widely used in recent years and has been shown to exhibit higher preference and usability over traditional interfaces ( Wonsick and Padir, 2020 ). VR also provides an opportunity to create more immersive and intuitive interfaces due to its capability of interacting and visualizing in 3D.…”

“…Unlike most virtual fixture methods of shared control relying on haptic guidance to enhance telepresence ( O’Malley et al, 2005 ; Selvaggio et al, 2018 ; Zeestraten et al, 2018 ), ours instead adopts a visually immersive experience through VR. Moreover, most teleoperation works in VR present a robot embodied view of the remote environment ( Xi et al, 2019 ; Hetrick et al, 2020 ; Wonsick and Padir, 2020 ). However, recent works have also shown the utility of model-based perspectives ( Van de Merwe et al, 2019 ; Wonsick et al, 2021a ).…”

An Approach Based on VR to Design Industrial Human-Robot Collaborative Workstations

“…The use of VR has the potential to simulate cooperative processes in advance, considering HRI from the design stages, and to include workers and their individual behavior into the simulation. Thanks to these benefits, there has been a significant increase in the use of VR in robotics in last years [12]. Moreover, VR simulations allow securing collaborative processes and reducing physical and mental barriers between humans and robots.…”