Towards a Vision-Based Concept for Gesture Control of a Robot Providing Visual Feedback

Gesture-based Human-Machine Interaction: Taxonomy, Problem Definition, and Analysis

“…Studies focusing only on continuous gestures are usually limited to reactions that are conceptually simple, e.g., the control of a mobile robot [21], or a one to one mapping between the human hand and the robot end-effector for tele-operation purposes [22], where the main problem to solve is geometric in nature. In order to attain more complex reactions, future works in the field should consider a principled integration between continuous and discrete gestures.…”

“…Sensor Number of Description gestures [13] Stereo camera 6 Static arm poses [14] Camera -Discrete hand and fingers motions [15] RGB camera 8 Bimanual static hand and fingers poses [16] Accelerometer 12 Discrete arm motions [17] Accelerometer 6 Discrete arm motions [18] RGB-D camera 5 Discrete arm motions [19] RGB-D camera 10 Discrete arm motions [20] RGB-D camera 1 Static hand pose [21] Accelerometer -Continuous arm motion [22] RGB-D camera -Continuous hand motion [23] RGB camera 8 Static hand and fingers poses [16] and providing commands to wheeled robots [17]. Arm gestures have been explored to provide task level information to a mobile manipulator [18], and to control a wheeled robot [19].…”

“…While pointing gestures have proved useful in indicating directions to assist a humanoid robot in navigation tasks [20], the usage of gestures to specify commands has been explored in difficult scenarios where other kinds of communication are very likely to fail, such as underwater [23]. The communication of a discrete command to an intelligent system implies the usage of a discrete gesture, while continuous commands used to tele-operate a wheeled robot [21] or a manipulator [22] necessitate continuous gestures 3 . Table I summarises the previously discussed papers, describing the used sensors, the number of gestures (if applicable), and the gesture types.…”

“…In certain cases, this is possible using raw data [21]. However, data should be processed to extract relevant, possibly semantic features, such as the 2D position of the hand with respect to an image plane [22]. As it is customary, we refer to this procedure as feature extraction.…”

“…Lack of direct feedback in HMI can ultimately lead to a lack of trust in the system and a deranged user experience [51]. Among the methods currently exploited to provide feedback to users, we can mention classical GUIs [22], haptics [65], augmented [66] and virtual [67] reality.…”