Realtime Hand-Object Interaction Using Learned Grasp Space for Virtual Environments

Abstract

We present a realtime virtual grasping algorithm to model interactions with virtual objects. Our approach is designed for multi-fingered hands and makes no assumptions about the motion of the user's hand or the virtual objects. Given a model of the virtual hand, we use machine learning and particle swarm optimization to automatically pre-compute stable grasp configurations for that object. The learning pre-computation step is accelerated using GPU parallelization. At runtime, we rely on the pre-computed stable grasp configurations, and dynamics/non-penetration constraints along with motion planning techniques to compute plausible looking grasps. In practice, our realtime algorithm can perform virtual grasping operations in less than 20ms for complex virtual objects, including high genus objects with holes. We have integrated our grasping algorithm with Oculus Rift HMD and Leap Motion controller and evaluated its performance for different tasks corresponding to grabbing virtual objects and placing them at arbitrary locations. Our user evaluation suggests that our virtual grasping algorithm can increase the user's realism and participation in these tasks and offers considerable benefits over prior interaction algorithms, such as pinch grasping and raycast picking.

Systematic Review on Realism Research Methodologies on Immersive Virtual, Augmented and Mixed Realities

“…Due to how the overlay works, virtual objects cannot completely occlude the real environment as they will always be transparent. [48], [49], [75], [76], [78] Custom User Preference 10 (11.1%) [36], [60], [67]- [69], [73], [86], [95], [97], [105] Presence Questionnaire (PQ) [57] Presence 7 [59], [60], [66], [74], [84], [ [43], [43], [56], [72], [96], [108], [113], [118], [119] Custom User Performance 7 (100%) [106], [112], [120]- [124] Simulator Sickness Questionnaire (SSQ) [125] Physiologic Responses 8 [56], [61], [66], [106], [110], [112], [126] Motion Sickness Assessment Questionnaire [127] Physiologic Responses 1 [109]…”

Freehand Grasping: An Analysis of Grasping for Docking Tasks in Virtual Reality

“…However, current approaches predominantly rely on computed grasps, not taking into account humans' grasping actions [41]. However, these limitations can be addressed by closely analysing human actions while grasping objects in VR [69]. Accordingly, Blaga et al [8] analysed human actions while grasping virtual objects by investigating the influence of thermal visual cues on grasping in VR, to understand how virtual environments influence grasping approaches.…”

Virtual hands in VR

“…Moving toward natural interface metaphors and generic manipulation with hands in VR, a promising path forward is the employment of physical simulation for hands, as in [81,137,48,114,18] among others. The key insight is that the force-based interaction afforded by physical simulation mimics the manner in which we affect the real world with our hands.…”

“…Intuitive activities in VR enhance engagement and interactivity that contribute to enriched user experiences. However, the development of interfaces that bridge the gap between real-world input and virtual interpretation remains challenging due to a number of factors [39,18,114,35]. Foremost, speed is key for real-time performance.…”