Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

Abstract

This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learningbased initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods. INDEX TERMS Deep learning, eye gazing, hands-free interaction, head gestures, human-robot interaction, mixed reality, object detection.

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

“…UIs and Widgets → Menus 115 Figure: [27,58] - [9-12, 18, 19, 24, 25, 27, 45, 48, 51-53, 55, 58, 61-63, 65, 66, 71, 71, 79, 89, 92, 92, 94, 96, 101, 103, 105, 112, 116, 120, 126, 131, 136, 137, 140, 144, 149, 150, 153, 156, 159, 168, 174, 181, 188, 190, 190, 194, 194-196, 196, 199, 206, 207, 215, 224, 228, 231, 233, 234, 239, 242, 253, 253, 262, 263, 269, 271, 274, 275, 278, 278, 286, 289, 292, 292, 297, 301, 301, 302, 305, 325, 326, 328, 333, 336, 338, 344, 351, 351, 357, 360, 373, 375, 377, 380, 381, 385, 387, 397, 400, 407, 410, 412, 412, 416, 420, 420, 429, 437, 443, 445, 456, 459, 465, 469, 479, 484, 487, 488, 488, 496] → Information Panels 80 Figure: [15,145] - [9, 15, 18, 44, 45, 48, 52-55, 59, 71, 86, 89, 92, 94, 96, 98, 104, 107, 112, 115, 120, 128, 136, 144, 145, 149, 156, 181, 185, 188, 190, 194-196, 199, 207, 223, 224, 228, 229, 232-234, 253, 262, 267, 278, 290, 292, 301, 315, 319, 329, 332, 336, 338, 344, 351, 355, 360, 373, 379-381, 412, 416, 420, 429, 437, 443, 451, 465, 470, 474, 479, 484, 488 163, 164, 174, 177, 181, 190, 190, 191, 194, 194, 196, 196, 199-201, 206, 212, 216, 220 166, 168, 169, 171, 174, 177, 181-185, 188, 189, 191, 194-196, 199-201, 206, 207, 209-211, 213, 215, 216, 220, 221, 226-234, 237-239, 241-243, 248, 251-253, 255-257, 262-269, 271-275, 278, 279, 283-288, 295-297, 301-303, 305, 306, 309, 310, 312, 313, 315, 317-320, 323, 325-327, 329, 330, 332-338, 341-344, 346, 347, 350-352, 358, 360-362, 365-368, 370, 372, 373, 375-377, 380, 382, 384, 385, 387-389, 391, 393, 395-397, 400, 402, 404, 406-409, 412, 413, 415, 417, 418, 431, 436, 437 136, 137, 139, 141, 156, 158, 161, 162, 165, 166, 168, 174, 181-183, 189, 194, 199, 200, 206, 211, 216, 225, 226, 231, 232, 237, 238, 242, 255, 257, 262, 270-274, 278, 285, 287, 296, 297, 301, 302, 320, 325-327, 329, 332, 336, 337, 341, 342, 352, 357, 358, 360, 362, 366, 368, 372, 375, 376, 387, 393, 396, 397, 400, 401, 410, 412, 431, 437, 439-441, 454, 465, 469, 474, 477, 488, 494, 496] → Gaze 14 Figure: [482] - [27,28,33,38,67,114,262,300,320,325,336,358,362,…”

Mixed, Augmented and Virtual, Reality Applied to the Teaching of Mathematics for Architects

“…An example thereof is the investigation undertaken by [48], who designed MR learning experiences in clinical practical training as a complement to theoretical training. Likewise, [49] proposed an original hands-free interaction method through multimodal gestures supported on deep learning in MR environments.…”

Adapt or Perish? Exploring the Effectiveness of Adaptive DoF Control Interaction Methods for Assistive Robot Arms

“…This made it difficult for them to predict how the robot would move, even with the visual cues provided by the arrows. To mitigate this problem, our concept might be combined with a "digital twin" of the robot arm, which demonstrates the movement virtually before the real robot performs it physically [31].…”