Multimodal data collection framework for mental stress monitoring

Kye, Saewon; Moon, Junhyung; Lee, Juneil; Choi, In-Ho; Cheon, Dongmi; Lee, Kyoungwoo

doi:10.1145/3123024.3125616

Cited by 12 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications have been seen in audio-visual speech recognition [52], image captioning [63], machine translation [34], sentiment analysis [55] and affect recognition [30]. In the space of ubiquitous computing, example applications include human activity recognition [1], sleep detection [12] and emotion recognition [36]. Many recognition tasks were previously only primarily performed with unimodal learning, with the availability of low-energy sensors, many such tasks are recently explored using multimodal learning.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

224

123

View full text Add to dashboard Cite

Wearables and mobile devices see the world through the lens of half a dozen low-power sensors, such as, barometers, accelerometers, microphones and proximity detectors. But differences between sensors ranging from sampling rates, discrete and continuous data or even the data type itself make principled approaches to integrating these streams challenging. How, for example, is barometric pressure best combined with an audio sample to infer if a user is in a car, plane or bike? Critically for applications, how successfully sensor devices are able to maximize the information contained across these multi-modal sensor streams often dictates the fidelity at which they can track user behaviors and context changes. This paper studies the benefits of adopting deep learning algorithms for interpreting user activity and context as captured by multi-sensor systems. Specifically, we focus on four variations of deep neural networks that are based either on fully-connected Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs). Two of these architectures follow conventional deep models by performing feature representation learning from a concatenation of sensor types. This classic approach is contrasted with a promising deep model variant characterized by modality-specific partitions of the architecture to maximize intra-modality learning. Our exploration represents the first time these architectures have been evaluated for multimodal deep learning under wearable data -- and for convolutional layers within this architecture, it represents a novel architecture entirely. Experiments show these generic multimodal neural network models compete well with a rich variety of conventional hand-designed shallow methods (including feature extraction and classifier construction) and task-specific modeling pipelines, across a wide-range of sensor types and inference tasks (four different datasets). Although the training and inference overhead of these multimodal deep approaches is in some cases appreciable, we also demonstrate the feasibility of on-device mobile and wearable execution is not a barrier to adoption. This study is carefully constructed to focus on multimodal aspects of wearable data modeling for deep learning by providing a wide range of empirical observations, which we expect to have considerable value in the community. We summarize our observations into a series of practitioner rules-of-thumb and lessons learned that can guide the usage of multimodal deep learning for activity and context detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

224

123

View full text Add to dashboard Cite

show abstract

“…To define σ , it is also necessary to determine for which value of the difference HR − HR 0 the subject can be considered sufficiently stressed or scared to justify a considerable decrease of the cobot speed; we refer to this value as HR . Experiments to evaluate the range of HR variability due to stress and fear were conducted with human participants watching scary movies 30 or pictures 31 , executing cognitive tasks 32 and interacting with robots [33][34][35] . The study of Weistroffer et al 34 , which studied the acceptability for human participants of the presence of robots in assembly lines, is the closest to our work.…”

Section: Hr-based Speed Modulationmentioning

confidence: 99%

Perceived safety in human–cobot interaction for fixed-path and real-time motion planning algorithms

Tusseyeva

Oleinikov

Sandygulova

et al. 2022

Sci Rep

View full text Add to dashboard Cite

This study investigates how different motion planning algorithms, implemented on a collaborative robot (cobot), are perceived by 48 human subjects. The four implemented algorithms ensure human safety based on the concept of speed and separation monitoring, but differ based on the following characteristics: (a) the cobot motion happens either along a fixed path or with a trajectory that is continuously planned in real time via nonlinear model predictive control, to increase cobot productivity; (b) the cobot speed is further reduced—or not—in real time based on heart rate measurements, to increase perceived safety. We conclude that (1) using a fixed path—compared to real-time motion planning—may reduce productivity and, at least when heart rate measurements are not used to modify the cobot speed, increases perceived safety; (2) reducing cobot speed based on heart rate measurements reduces productivity but does not improve perceived safety; (3) perceived safety is positively affected by habituation during the experiment, and unaffected by previous experience.

show abstract

“…Using wearable devices in AfC is a case that many researchers have become interested in recently, due to the many possible applications, such as ambient assisted living [12] or social analysis of a group of people [13], among other applications, in areas ranging from validation between such devices and medical-quality platforms [14,15,16], through emotion detection and recognition [12,14,17,18,19,20,21], to creating new apparatuses that enable unobtrusive data acquisition [16,22]; the last approach being an answer to the remarks presented in [8]. It is worth mentioning that, in terms of AfG, sensors incorporated into gloves and shoes or gaming equipment (such as the mouse or game pad), are much more appropriate for active gameplay.…”

Section: Introductionmentioning

confidence: 99%

Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants

Nalepa

Kutt

Giżycka

et al. 2019

Sensors

View full text Add to dashboard Cite

In this paper, we consider the use of wearable sensors for providing affect-based adaptation in Ambient Intelligence (AmI) systems. We begin with discussion of selected issues regarding the applications of affective computing techniques. We describe our experiments for affect change detection with a range of wearable devices, such as wristbands and the BITalino platform, and discuss an original software solution, which we developed for this purpose. Furthermore, as a test-bed application for our work, we selected computer games. We discuss the state-of-the-art in affect-based adaptation in games, described in terms of the so-called affective loop. We present our original proposal of a conceptual design framework for games, called the affective game design patterns. As a proof-of-concept realization of this approach, we discuss some original game prototypes, which we have developed, involving emotion-based control and adaptation. Finally, we comment on a software framework, that we have previously developed, for context-aware systems which uses human emotional contexts. This framework provides means for implementing adaptive systems using mobile devices with wearable sensors.

show abstract

Multimodal data collection framework for mental stress monitoring

Cited by 12 publications

References 10 publications

Multimodal Deep Learning for Activity and Context Recognition

Multimodal Deep Learning for Activity and Context Recognition

Perceived safety in human–cobot interaction for fixed-path and real-time motion planning algorithms

Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants

Contact Info

Product

Resources

About