Activity Recognition Based on Millimeter-Wave Radar by Fusing Point Cloud and Range–Doppler Information

Abstract

Millimeter-wave radar has demonstrated its high efficiency in complex environments in recent years, which outperforms LiDAR and computer vision in human activity recognition in the presence of smoke, fog, and dust. In previous studies, researchers mostly analyzed either 2D (3D) point cloud or range–Doppler information from radar echo to extract activity features. In this paper, we propose a multi-model deep learning approach to fuse the features of both point clouds and range–Doppler for classifying six activities, i.e., boxing, jumping, squatting, walking, circling, and high-knee lifting, based on a millimeter-wave radar. We adopt a CNN–LSTM model to extract the time-serial features from point clouds and a CNN model to obtain the features from range–Doppler. Then we fuse the two features and input the fused feature into the full connected layer for classification. We built a dataset based on a 3D millimeter-wave radar from 17 volunteers. The evaluation result based on the dataset shows that this method has higher accuracy than utilizing the two kinds of information separately and achieves a recognition accuracy of 97.26%, which is about 1% higher than other networks with only one kind of data as input.

Orientation-Independent Human Activity Recognition Using Complementary Radio Frequency Sensing

“…Moreover, we can notice that the Radar I classifier was able to classify each activity with reasonable precision and recall, irrespective of the strategy used to split the Radar I feature vectors into training and test sets. However, comparing the results of the Radar I classifier with other existing works on mmWave radar-based HAR [15,[20][21][22][23][24][25], it can be argued that the recognition accuracy of the Radar I classifier is at the lower end. This is due to the fact that the Radar I classifier attempted to recognize orientation-independent activities, unlike other approaches that recognized activities performed along the 0°aspect angle.…”

“…In recent years, the use of mmWave radar as an RF sensing technology has increased in various human-centric applications, such as HAR [15,[20][21][22]24,25]; gesture recognition [28][29][30]; gait recognition [31,32]; human step counting [4]; fall detection [23,33]; and sign language gesture recognition [34]. In general, the tasks of recognizing human activities, (sign language) gestures, and detecting accidental falls using mmWave radar follow similar steps.…”

“…The proposed approach could classify seven activities with 98% accuracy and detect falls with 97.61% accuracy. Another work [15] fused the features extracted from the point cloud and the range-Doppler information to classify six different human activities, i.e., "boxing, jumping, squatting, walking in place, high knee lifting, and circling in place". They used a CNN-LSTM network to extract features from 3D point cloud data and a CNN network to extract features from the range-Doppler profile.…”

“…In recent years, RF-based approaches for HAR have attracted a lot of research interest due to their advantages over wearable sensor-or vision-based approaches for HAR. For example, they offer high environmental adaptability and can recognize human activities without risking privacy and comfort [15]. Wi-Fi [2,[16][17][18][19] and mmWave radar [15,[20][21][22][23][24][25] are the most widely used sensing technologies within the context of RFbased HAR systems.…”

“…For example, they offer high environmental adaptability and can recognize human activities without risking privacy and comfort [15]. Wi-Fi [2,[16][17][18][19] and mmWave radar [15,[20][21][22][23][24][25] are the most widely used sensing technologies within the context of RFbased HAR systems. Note that the term mmWave radar is generally used to refer to radars that use short-wavelength electromagnetic waves.…”