Real-time dynamic gesture recognition system based on depth perception for robot navigation

Abstract

Natural human robot interaction based on the dynamic hand gesture is becoming a popular research topic in the past few years. The traditional dynamic gesture recognition methods are usually restricted by the factors of illumination condition, varying color and cluttered background. The recognition performance can be improved by using the hand-wearing devices but this is not a natural and barrier-free interaction. To overcome these shortcomings, the depth perception algorithm based on the Kinect depth sensor is introduced to carry out 3D hand tracking. We propose a novel start/end point detection method for segmenting the 3D hand gesture from the hand motion trajectory. Then Hidden Markov Models (HMMs) are implemented to model and classify the hand gesture sequences and the recognized gestures are converted to control commands for the interaction with the robot. Seven different hand gestures performed by two hands can sufficiently navigate the robot. Experiments show that the proposed dynamic hand gesture interaction system can work effectively in the complex environment and in real-time with an average recognition rate of 98.4%. And further experiments for the robot navigation also verify the robustness of our system.

Survey on 3D Hand Gesture Recognition

“…The hand position can be either determined by hand segmentation [137] or the body skeleton generated by NITE middleware with the Kinect sensor [138]. In addition to the palm position, the orientation or angle of the hand centroid in a 3D hand gesture trajectory can also be used [46]. Miranda et al [139] described the pose in each frame using a tailored angular representation of the skeleton joints.…”

“…It is very suitable to classify the tracking based gestures such as trajectory-based signed digits [138]. HMM was also applied to the robotic navigation [46] and sign language recognition [141,144]. Compared to HMM, the main advantage of DTW is that it can automatically align the sequences which have different lengths and return the proper distance.…”

Attribute Guided Dictionary Learning

“…Attributes can also be also applied into action recognition. Instead of employing 3D models [20] or body segmentation method [21], Qiu et al [14] proposed learning a compact dictionary for action recognition, in which every basis is regarded as an action attribute. However, all these attributes have no semantic meanings, and thus can hardly be generalized to a novel action.…”

Dynamic gesture recognition using 3D trajectory

“…In [4], a stereo vision-based platform is constructed, which acquired the trajectory of the dynamic gesture by detecting handheld retro-reflective markers. With the emergence and the application of the Kinect [7] and leap motion [8], the 3D information of the gesture can be recorded. In our system, the sequence points of the position of palm is captured by the leap motion sensor, which depends on the built-in two cameras to capture images from different angles, the reconstruction of the palm in the real world of three-dimensional space motion information.…”

“…Hidden Markov Model (HMM) [13] is one of the classic mathematical methods and is widely used to classify the sequence data [1,3,7,15]. In this paper, we modified the topology and the traditional initial value of state transition probability matrix to make the system more robust.…”