Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks

Sano, Akane; Chen, Weixuan; Lopez-Martinez, Daniel; Taylor, Sara; Picard, Rosalind W.

doi:10.1109/jbhi.2018.2867619

Cited by 49 publications

(21 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Narayanan et al [33] designed the gate recurrent cell with the multimodal sensor data to model driver behaviors. Sano et al [34] proposed a multimodal BiLSTM to detect ambulatory sleep in which the BiLSTM was used to extract the features of the data collected from wearable devices. Then each intermodality feature was concatenated by a fully connected network.…”

Section: B Data Fusion Methodsmentioning

confidence: 99%

Optimizing Features Quality: A Normalized Covariance Fusion Framework for Skeleton Action Recognition

Huang

Yan

2020

IEEE Access

View full text Add to dashboard Cite

Section: B Data Fusion Methodsmentioning

confidence: 99%

Optimizing Features Quality: A Normalized Covariance Fusion Framework for Skeleton Action Recognition

Huang

Yan

2020

IEEE Access

View full text Add to dashboard Cite

“…Narayanan, Siravuru, and Dariush (2019) designed the gate recurrent cell with the multimodal sensor data to model driver behaviors. Sano, Chen, Lopez-Martinez, Taylor, and Picard (2019) proposed a multimodal BiLSTM to detect ambulatory sleep in which the BiLSTM is used to extract features of data collected from wearable devices. Then each intermodality feature is concatenated by a fully connected network.…”

Section: Examplementioning

confidence: 99%

A Survey on Deep Learning for Multimodal Data Fusion

et al. 2020

View full text Add to dashboard Cite

With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.

show abstract

“…They used GRU to generate a description of variable length from a given image. Similarly, Sano et al [222] proposed a multimodal BiLSTM for ambulatory sleep detec- tion. In this case, BiLSTM was used to extract features from the wearable device and synthesize temporal information.…”

Section: Recurrent Neural Network Basedmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks

Cited by 49 publications

References 45 publications

Optimizing Features Quality: A Normalized Covariance Fusion Framework for Skeleton Action Recognition

Optimizing Features Quality: A Normalized Covariance Fusion Framework for Skeleton Action Recognition

A Survey on Deep Learning for Multimodal Data Fusion

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Contact Info

Product

Resources

About