Multimodal feature fusion for CNN-based gait recognition: an empirical comparison

Abstract

People identification in video based on the way they walk (i.e. gait) is a relevant task in computer vision using a non-invasive approach. Standard and current approaches typically derive gait signatures from sequences of binary energy maps of subjects extracted from images, but this process introduces a large amount of non-stationary noise, thus, conditioning their efficacy. In contrast, in this paper we focus on the raw pixels, or simple functions derived from them, letting advanced learning techniques to extract relevant features. Therefore, we present a comparative study of different Convolutional Neural Network (CNN) architectures on three low-level features (i.e. gray pixels, optical flow channels and depth maps) on two widely-adopted and challenging datasets: TUM-GAID and CASIA-B. In addition, we perform a comparative study between different early and late fusion methods used to combine the information obtained from each kind of low-level features. Our experimental results suggest that (i ) the use of hand-crafted energy maps (e.g. GEI) is not necessary, since equal or better results can be achieved from the raw pixels; (ii ) the combination of multiple modalities (i.e. gray pixels, optical flow and depth maps) from different CNNs allows to obtain state-of-the-art results on the gait task with an image resolution several times smaller than the previously reported results; and, (iii ) the selection of the architecture is a critical point that can make the difference between state-of-the-art results or poor results. He et al.[10] proposed a new kind of CNN, named ResNet, which has a large number of convolutional layers and 'residual connections' to avoid the vanishing gradient problem.Although several papers can be found for the task of human action recognition using DL techniques, few works apply DL to the problem of gait recognition. In [22], Hossain and Chetty propose the use of Restricted Boltzmann Machines to extract gait features from binary silhouettes, but a very small probe set (i.e. only ten different subjects) were used for validating their approach. A more recent work, [23], uses a random set of binary silhouettes of a sequence to train a CNN that accumulates the calculated features in order to achieve a global representation of the dataset. In [24], raw 2D GEI are employed to train an ensemble of CNN, where a Multilayer Perceptron (MLP) is used as classifier. Similarly, in [25] a multilayer CNN is trained with GEI data. A novel approach based on GEI is developed on [8], where the CNN is trained with pairs of gallery-probe samples and using a distance metric. Castro et al.[26] use optical flow obtained from raw data frames. An in-dept evaluation of different CNN architectures based on optical flow maps is presented in [27]. Finally, in [28] a multitask CNN with a combined loss function with multiple kind of labels is presented.Despite most CNNs are trained with visual data (e.g. images or videos), there are some works that build CNNs for different kinds of data like inertial sensors or human skel...

Improved Deep Learning Technique to Detect Freezing of Gait in Parkinson’s Disease Based on Wearable Sensors

“…For example, Handojoseno et al used the wavelet coefficients of electroencephalogram (EEG) signals as the input of the Multilayer Perceptron Neural Network and k-Nearest Neighbor classifier, which can predict the transition from walking to FOG with 87% sensitivity and 73% accuracy [14]. The second group generally uses three-dimensional (3D) motion analysis systems [15][16][17][18], plantar pressure measurement systems [19][20][21] or inertial sensors (accelerometers, gyroscopes or magnetometers) [22][23][24][25] to obtain more intuitive gait kinematics or dynamics signals. Delval et al used multiple cameras to capture the gait kinematics signals of patients with reflective markers attached to their bodies from different angles [17], and Okuno et al used a plantar pressure measurement system (1.92 × 0.88 m) to record the soles of the patients walking [19].…”

Review of gait recognition approaches and their challenges on view changes

“…In [70], four different CNN architectures are proposed for gait feature extraction, including (i) 2D CNN: linear CNN with four 2D convolutions, (ii) 3D CNN: 3D CNN with four 3D convolutions, (iii) ResNet‐A: residual CNN with a 2D convolution and three residual blocks, and (iv) ResNet‐B: residual CNN with a 2D convolution and four residual blocks. Three different modalities are separately attempted in the CNN model, including grey pixels, optical flow channels, and depth maps.…”

Multimodal Gait Recognition Under Missing Modalities

“…Kinect, mobiles) or techniques that produce different kinds of data like depth [7] or optical flow [8]. Thus, some works exploit multimodality and show that it leads to better representations and improved results [2,9,10,11,12]. Nevertheless, their common limitation is their inability to handle missing modalities.…”