UniTS

Li, Shuheng; Chowdhury, Ranak Roy; Shang, Jingbo; Gupta, Rajesh K.; Hong, Dezhi

doi:10.1145/3485730.3485942

Cited by 14 publications

(8 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, it is possible to substitute missing data with a default value, then use SADeepSense to perform HAR, tackling both data quality issues. UniTS [21] proposes the short-time Fourier series-inspired neural network, named TS-Encoder, and employs multiple TS-Encoders to extract information in the time and frequency domains at various scales for classification tasks. Segmenting sensor data using a larger window is more favorable for UniTS as it can fully exploit the multi-scale information.…”

Section: 23mentioning

confidence: 99%

“…For pre-processing, 45 sensor channels and windows of 256 timestamps are used to perform a 4-class human locomotion recognition on the OPPORTUNITY activity recognition dataset [31]. Our work differs from UniTS [21] in that we consider consecutive missing and noisy data simultaneously, using a stochastic corruption process described in Section 3.2, while they assume each sample may be missing with a probability that is independent of the other samples, hence long periods of missing data are very unlikely in their work. Furthermore, our proposed model achieves higher classification accuracy and F1 score in the presence of these issues (see Section 7).…”

Section: 23mentioning

confidence: 99%

“…UniTS [21]. It is a robust neural network architecture that can learn from multimodal data with artificially injected noise and dropped data points.…”

Section: Human Activity Recognition Baselinesmentioning

confidence: 99%

“…As a result, most related work that considers data quality issues handles either missing data [9,12,36,37,41] or noisy measurements [14,43,44] only. To our knowledge, UniTS [21] is presently the only multimodal fusion model that is designed to be robust to both consecutive missing data and noise. To train this model, the authors add Gaussian noise to training data, and randomly mask readings for simulating the effect of missing data.…”

Section: Introductionmentioning

confidence: 99%

“…The proposed HAR model is compared with 4 baselines that are shown to have superior performance over several multimodal fusion models in the HAR task. [21]. • We conduct thorough evaluation of Centaur on three multimodal HAR datasets, namely PAMAP2, OPPORTUNITY, and HHAR, to study its robustness to sensor faults that might occur simultaneously.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Robust Multimodal Fusion for Human Activity Recognition

Xaviar¹,

Yang²,

Ardakanian³

2023

Preprint

View full text Add to dashboard Cite

The proliferation of IoT and mobile devices equipped with heterogeneous sensors has enabled new applications that rely on the fusion of time-series data generated by multiple sensors with different modalities. While there are promising deep neural network architectures for multimodal fusion, their performance falls apart quickly in the presence of consecutive missing data and noise across multiple modalities/sensors, the issues that are prevalent in realworld settings. We propose Centaur, a multimodal fusion model for human activity recognition (HAR) that is robust to these data quality issues. Centaur combines a data cleaning module, which is a denoising autoencoder with convolutional layers, and a multimodal fusion module, which is a deep convolutional neural network with the self-attention mechanism to capture cross-sensor correlation. We train Centaur using a stochastic data corruption scheme and evaluate it on three datasets that contain data generated by multiple inertial measurement units. We show that Centaur's data cleaning module outperforms two state-of-the-art autoencoder-based architectures, and its multimodal fusion module outperforms four strong baselines. Compared to two robust fusion architectures from the related work, Centaur is more robust especially to consecutive missing data that occur in multiple sensor channels, achieving on average 11.59-17.52% higher accuracy in the HAR task.

show abstract

Section: 23mentioning

confidence: 99%

Section: 23mentioning

confidence: 99%

“…UniTS [21]. It is a robust neural network architecture that can learn from multimodal data with artificially injected noise and dropped data points.…”

Section: Human Activity Recognition Baselinesmentioning

confidence: 99%