Motion Pattern Recognition in 4D Point Clouds

Salami, Dariush; Palipana, Sameera; Kodali, Manila; Sigg, Stephan

doi:10.1109/mlsp49062.2020.9231569

Cited by 13 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Table 2, the performance of Tesla and Tesla-V on Pantomime dataset is compared to baseline models of PointNet [16], PointNet++ [52], O&H [56], PointGest [19], RadHar [45], PointLSTM [17], Pantomime [18], and DEC [24]. In PointNet, PointNet++, and DEC, the frames are aggregated through time dimension into a single frame representing the whole gesture, since they are designed to classify static point clouds.…”

Section: Overall Results On Pantomime Datasetmentioning

confidence: 99%

“…Previous attempts at capturing spatio-temporal features of dynamic point clouds include using a combination of RNN with either 3DCNN or PointNet layers [18], [19], [56], as well as using a modified RNN layer to propagate information temporally while preserving the spatial structure in each frame [17]. In real-world applications, these models are constrained by their high computational complexity and restricted generalizability on point clouds generated in different settings.…”

Section: Dynamic Point Cloudsmentioning

confidence: 99%

“…The input representation plays an important role in both accuracy and time-complexity of deep learning based systems. RGB images [12], [13], [14], depth images [12], [13], [14], spectrograms of Doppler signals [15], and point clouds [16], [17], [18], [19] are commonly used representations. Among these, point clouds, i.e., unordered sets of points in space, are the standard output of a wide range of sensors [16].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds

Salami,

Hasibi,

Palipana

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy while reducing the computational complexity and, hence, the execution time. In particular, the approach, is able to predict a gesture almost 8 times faster than the most accurate competitor. Our performance evaluation in different scenarios (environments, angles, distances) shows that Tesla generalizes well and improves the accuracy up to 20% in challenging scenarios like a through-wall setting and sensing at extreme angles. Utilizing Tesla, we develop Tesla-Rapture, a real-time implementation using a mmWave Radar on a Raspberry PI 4 and evaluate its accuracy and time-complexity. We also publish the source code, the trained models, and the implementation of the model for embedded devices.

show abstract

Section: Overall Results On Pantomime Datasetmentioning

confidence: 99%

Section: Dynamic Point Cloudsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds

Salami,

Hasibi,

Palipana

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…as well as the cubic increase in the computational complexity are two major drawbacks of this group. Finally, direct point-cloud processing gesture recognition models are another group of approaches which attempt to classify gestures directly from point-clouds [11,4,12]. These models have addressed the need for data conversion and its computational-complexity and information loss problems.…”

Section: Feature Vectormentioning

confidence: 99%

Zero-Shot Motion Pattern Recognition from 4D Point-Clouds

Salami

Sigg

2021

2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)

Self Cite

View full text Add to dashboard Cite

We address a timely and relevant problem in signal processing: The recognition of patterns from spatial data in motion through a zero-shot learning scenario. We introduce a neural network architecture based on Siamese networks to recognize unseen classes of motion patterns. The approach uses a graph-based technique to achieve permutation invariance and also encodes moving point clouds into a representation space in a computationally efficient way. We evaluated the model on an open dataset with twenty-one gestures. The model outperformes state-of-the-art architectures with a considerable margin in four different settings in terms of accuracy while reducing the computational complexity up to 60 times.

show abstract

“…A temporal point-cloud is a sequence of frames through time, each of which consists of an unordered set of points [21], [22]. Fuelled by commercial availability of point cloud generating mmWave radars, mmWave radar point cloud based sensing has been actively investigated for e.g.…”

Section: Related Workmentioning

confidence: 99%

Integrating Sensing and Communication in Cellular Networks via NR Sidelink

Salami,

Hasibi,

Savazzi

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

RF-sensing, the analysis and interpretation of movement or environment-induced patterns in received electromagnetic signals, has been actively investigated for more than a decade. Since electromagnetic signals, through cellular communication systems, are omnipresent, RF sensing has the potential to become a universal sensing mechanism with applications in smart home, retail, localization, gesture recognition, intrusion detection, etc. Specifically, existing cellular network installations might be dual-used for both communication and sensing. Such communications and sensing convergence is envisioned for future communication networks. We propose the use of NR-sidelink direct device-to-device communication to achieve device-initiated, flexible sensing capabilities in beyond 5G cellular communication systems. In this article, we specifically investigate a common issue related to sidelink-based RF-sensing, which is its angle and rotation dependence. In particular, we discuss transformations of mmWave point-cloud data which achieve rotational invariance, as well as distributed processing based on such rotational invariant inputs, at angle and distance diverse devices. To process the distributed data, we propose a graph based encoder to capture spatio-temporal features of the data and propose four approaches for multi-angle learning. The approaches are compared on a newly recorded and openly available dataset comprising 15 subjects, performing 21 gestures which are recorded from 8 angles.

show abstract

Motion Pattern Recognition in 4D Point Clouds

Cited by 13 publications

References 10 publications

Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds

Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds

Zero-Shot Motion Pattern Recognition from 4D Point-Clouds

Integrating Sensing and Communication in Cellular Networks via NR Sidelink

Contact Info

Product

Resources

About