Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors

Bernal, Edgar A.; Yang, Xitong; Li, Qun; Kumar, J. R. Harish; Madhvanath, Sriganesh; Ramesh, Palghat; Bala, Raja

doi:10.1109/tmm.2017.2726187

Cited by 62 publications

(33 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the recent popularity of deep neural networks (DNN) in multimedia applications, several deep learning based fusion frameworks for HAR has recently been presented. In [23] a supervised deep multimodal fusion framework for process monitoring and verification in the medical and healthcare fields is presented that depends on simultaneous processing of motion data acquired with wearable sensors and video data acquired with body-mounted camera. Authors in [24] proposed DNN based fusion of images and inertial data for improving the performance of human action recognition.…”

Section: Related Workmentioning

confidence: 99%

Human Action Recognition Using Deep Multilevel Multimodal (${M}^{2}$ ) Fusion of Depth and Inertial Sensors

Ahmad

Khan

2020

IEEE Sensors J.

View full text Add to dashboard Cite

Multimodal fusion frameworks for Human Action Recognition (HAR) using depth and inertial sensor data have been proposed over the years. In most of the existing works, fusion is performed at a single level (feature level or decision level), missing the opportunity to fuse rich mid-level features necessary for better classification. To address this shortcoming, in this paper, we propose three novel deep multilevel multimodal (M 2 ) fusion frameworks to capitalize on different fusion strategies at various stages and to leverage the superiority of multilevel fusion. At input, we transform the depth data into depth images called sequential front view images (SFIs) and inertial sensor data into signal images. Each input modality, depth and inertial, is further made multimodal by taking convolution with the Prewitt filter. Creating "modality within modality" enables further complementary and discriminative feature extraction through Convolutional Neural Networks (CNNs). CNNs are trained on input images of each modality to learn low-level, high-level and complex features. Learned features are extracted and fused at different stages of the proposed frameworks to combine discriminative and complementary information. These highly informative features are served as input to a multi-class Support Vector Machine (SVM). We evaluate the proposed frameworks on three publicly available multimodal HAR datasets, namely, UTD Multimodal Human Action Dataset (MHAD), Berkeley MHAD, and UTD-MHAD Kinect V2. Experimental results show the supremacy of the proposed fusion frameworks over existing methods.Index Terms-Canonical correlation analysis, fusion of depth and inertial sensors, human action recognition,, multimodal fusion.

show abstract

Section: Related Workmentioning

confidence: 99%

Human Action Recognition Using Deep Multilevel Multimodal (${M}^{2}$ ) Fusion of Depth and Inertial Sensors

Ahmad

Khan

2020

IEEE Sensors J.

View full text Add to dashboard Cite

show abstract

“…Different ways of combining the features have been proposed, depending on the type of application, so that these vectors with multidimensional characteristics can then be morphed into transformed vectors of joint characteristics, from which classification is then carried out [24]. Examples of feature-level fusion are: Feature Aggregation [14,[25][26][27][28][29][30][31], Temporal Fusion [32], Support Vector Machine (SVM)-based multisensor fusion algorithm [33], and Data Fusion Location algorithm [18].…”

Section: State Of the Artmentioning

confidence: 99%

“…Coordinates x and y are width and height dimensions of the face image, and z is the depth dimension. The face points correspond to the left eye (0-7), right eye (8)(9)(10)(11)(12)(13)(14)(15), left eyebrow (16)(17)(18)(19)(20)(21)(22)(23)(24)(25), right eyebrow (26)(27)(28)(29)(30)(31)(32)(33)(34)(35), nose (36)(37)(38)(39)(40)(41)(42)(43)(44)(45)(46)(47), mouth (48)(49)(50)(51)(52)(53)(54)(55)(56)(57)…”

Section: Gfe Data Setsmentioning

confidence: 99%

Choosing the Best Sensor Fusion Method: A Machine-Learning Approach

Brena

Aguileta

Trejo

et al. 2020

Sensors

View full text Add to dashboard Cite

Multi-sensor fusion refers to methods used for combining information coming from several sensors (in some cases, different ones) with the aim to make one sensor compensate for the weaknesses of others or to improve the overall accuracy or the reliability of a decision-making process. Indeed, this area has made progress, and the combined use of several sensors has been so successful that many authors proposed variants of fusion methods, to the point that it is now hard to tell which of them is the best for a given set of sensors and a given application context. To address the issue of choosing an adequate fusion method, we recently proposed a machine-learning data-driven approach able to predict the best merging strategy. This approach uses a meta-data set with the Statistical signatures extracted from data sets of a particular domain, from which we train a prediction model. However, the mentioned work is restricted to the recognition of human activities. In this paper, we propose to extend our previous work to other very different contexts, such as gas detection and grammatical face expression identification, in order to test its generality. The extensions of the method are presented in this paper. Our experimental results show that our extended model predicts the best fusion method well for a given data set, making us able to claim a broad generality for our sensor fusion method.

show abstract

“…Summarizing such salient scenes is essential because FPV videos tend to be redundant [2]. However, FPV videos are unstable and noisy compared to third-person view (TPV) videos, and most existing methods of video summarization mainly focus on handling the stable scenes in a TPV video [3]- [19]. Moreover, the following differences between FPV and TPV videos substantially complicate summarizing FPV videos compared to summarizing TPV videos.…”

Section: Introductionmentioning

confidence: 99%

Multi-Sensor Integration for Key-Frame Extraction From First-Person Videos

Kanemura

Asoh

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Key-frame extraction for first-person vision (FPV) videos is a core technology for selecting important scenes and memorizing impressive life experiences in our daily activities. The difficulty of selecting key frames is the scene instability caused by head-mounted cameras used for capturing FPV videos. Because head-mounted cameras tend to frequently shake, the frames in an FPV video are noisier than those in a third-person vision (TPV) video. However, most existing algorithms for key-frame extraction mainly focus on handling the stable scenes in TPV videos. The technical development of key-frame extraction techniques for noisy FPV videos is currently immature. Moreover, most key-frame extraction algorithms mainly use visual information from FPV videos, even though our visual experience in daily activities is associated with human motions. To incorporate the features of dynamically changing scenes in FPV videos into our methods, integrating motions with visual scenes is essential. In this paper, we propose a novel key-frame extraction method for FPV videos that uses multi-modal sensor signals to reduce noise and detect salient activities via projecting multi-modal sensor signals onto a common space by canonical correlation analysis (CCA). We show that the two proposed multi-sensor integration models for key-frame extraction (a sparse-based model and a graph-based model) work well on the common space. The experimental results obtained using various datasets suggest that the proposed key-frame extraction techniques improve the precision of extraction and the coverage of entire video sequences.

show abstract

Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors

Cited by 62 publications

References 57 publications

Human Action Recognition Using Deep Multilevel Multimodal (${M}^{2}$ ) Fusion of Depth and Inertial Sensors

Human Action Recognition Using Deep Multilevel Multimodal (${M}^{2}$ ) Fusion of Depth and Inertial Sensors

Choosing the Best Sensor Fusion Method: A Machine-Learning Approach

Multi-Sensor Integration for Key-Frame Extraction From First-Person Videos

Contact Info

Product

Resources

About