Multimodal multisensor activity annotation tool

Barz, Michael; Moniri, Mohammad Mehdi; Weber, Markus; Sonntag, Daniel

doi:10.1145/2968219.2971459

Cited by 14 publications

(10 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications have been seen in audio-visual speech recognition [52], image captioning [63], machine translation [34], sentiment analysis [55] and affect recognition [30]. In the space of ubiquitous computing, example applications include human activity recognition [1], sleep detection [12] and emotion recognition [36]. Many recognition tasks were previously only primarily performed with unimodal learning, with the availability of low-energy sensors, many such tasks are recently explored using multimodal learning.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

225

123

View full text Add to dashboard Cite

Wearables and mobile devices see the world through the lens of half a dozen low-power sensors, such as, barometers, accelerometers, microphones and proximity detectors. But differences between sensors ranging from sampling rates, discrete and continuous data or even the data type itself make principled approaches to integrating these streams challenging. How, for example, is barometric pressure best combined with an audio sample to infer if a user is in a car, plane or bike? Critically for applications, how successfully sensor devices are able to maximize the information contained across these multi-modal sensor streams often dictates the fidelity at which they can track user behaviors and context changes. This paper studies the benefits of adopting deep learning algorithms for interpreting user activity and context as captured by multi-sensor systems. Specifically, we focus on four variations of deep neural networks that are based either on fully-connected Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs). Two of these architectures follow conventional deep models by performing feature representation learning from a concatenation of sensor types. This classic approach is contrasted with a promising deep model variant characterized by modality-specific partitions of the architecture to maximize intra-modality learning. Our exploration represents the first time these architectures have been evaluated for multimodal deep learning under wearable data -- and for convolutional layers within this architecture, it represents a novel architecture entirely. Experiments show these generic multimodal neural network models compete well with a rich variety of conventional hand-designed shallow methods (including feature extraction and classifier construction) and task-specific modeling pipelines, across a wide-range of sensor types and inference tasks (four different datasets). Although the training and inference overhead of these multimodal deep approaches is in some cases appreciable, we also demonstrate the feasibility of on-device mobile and wearable execution is not a barrier to adoption. This study is carefully constructed to focus on multimodal aspects of wearable data modeling for deep learning by providing a wide range of empirical observations, which we expect to have considerable value in the community. We summarize our observations into a series of practitioner rules-of-thumb and lessons learned that can guide the usage of multimodal deep learning for activity and context detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Deep Learning for Activity and Context Recognition

Radu

Tong

Bhattacharya

et al. 2018

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

225

123

View full text Add to dashboard Cite

show abstract

“…feature extraction). Indeed, Barz et al [ 19 ] highlight that most data acquisition and annotation tools are mostly limited to a particular sensor. This can be attributed to the fact that it seems to be necessary to consider different techniques or feature sets for different kinds of sensors.…”

Section: Related Workmentioning

confidence: 99%

Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets

Diete

Sztyler

Stuckenschmidt

2018

Sensors

View full text Add to dashboard Cite

Working with multimodal datasets is a challenging task as it requires annotations which often are time consuming and difficult to acquire. This includes in particular video recordings which often need to be watched as a whole before they can be labeled. Additionally, other modalities like acceleration data are often recorded alongside a video. For that purpose, we created an annotation tool that enables to annotate datasets of video and inertial sensor data. In contrast to most existing approaches, we focus on semi-supervised labeling support to infer labels for the whole dataset. This means, after labeling a small set of instances our system is able to provide labeling recommendations. We aim to rely on the acceleration data of a wrist-worn sensor to support the labeling of a video recording. For that purpose, we apply template matching to identify time intervals of certain activities. We test our approach on three datasets, one containing warehouse picking activities, one consisting of activities of daily living and one about meal preparations. Our results show that the presented method is able to give hints to annotators about possible label candidates.

show abstract

“…The segmentation and annotation of time-series can be performed using graphical user interfaces (GUIs) [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. Although some existing GUIs process similar data with similar goals, for example the annotation of human activities from videos and wearable sensors, they have been developed independently of each other [ 15 , 16 , 17 , 18 , 19 ]. This means the existing code is not re-used within the community, and thus, similar functionalities are implemented in numerous ways, leading to duplicated work.…”

Section: Introductionmentioning

confidence: 99%

“…One of the reasons for code not being re-used includes code not being publicly available [ 15 , 16 , 17 , 18 ]. According to a study conducted by Stodden et al [ 21 ], the major reason for not publishing code is that researchers declare their code as not being cleaned up and undocumented.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, it can be expected that developers of GUIs for time-series analysis would prefer Python as a programming language and therefore discard available GUIs that have not been written in Python. Furthermore, most of the GUIs seem not to be adaptable to other data formats and/or algorithms and, thus, can only be used with the data and algorithms from the original publication [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MaD GUI: An Open-Source Python Package for Annotation and Analysis of Time-Series Data

Ollenschläger

Küderle

Mehringer

et al. 2022

Sensors

View full text Add to dashboard Cite

Developing machine learning algorithms for time-series data often requires manual annotation of the data. To do so, graphical user interfaces (GUIs) are an important component. Existing Python packages for annotation and analysis of time-series data have been developed without addressing adaptability, usability, and user experience. Therefore, we developed a generic open-source Python package focusing on adaptability, usability, and user experience. The developed package, Machine Learning and Data Analytics (MaD) GUI, enables developers to rapidly create a GUI for their specific use case. Furthermore, MaD GUI enables domain experts without programming knowledge to annotate time-series data and apply algorithms to it. We conducted a small-scale study with participants from three international universities to test the adaptability of MaD GUI by developers and to test the user interface by clinicians as representatives of domain experts. MaD GUI saves up to 75% of time in contrast to using a state-of-the-art package. In line with this, subjective ratings regarding usability and user experience show that MaD GUI is preferred over a state-of-the-art package by developers and clinicians. MaD GUI reduces the effort of developers in creating GUIs for time-series analysis and offers similar usability and user experience for clinicians as a state-of-the-art package.

show abstract

Multimodal multisensor activity annotation tool

Cited by 14 publications

References 3 publications

Multimodal Deep Learning for Activity and Context Recognition

Multimodal Deep Learning for Activity and Context Recognition

Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets

MaD GUI: An Open-Source Python Package for Annotation and Analysis of Time-Series Data

Contact Info

Product

Resources

About