Cosmo

Ouyang, Xiaomin; Shuai, Xian; Zhou, Jiayu; Shi, Ivy Wang; Xie, Zhiyuan; Xing, Guoliang; Huang, Jianwei

doi:10.1145/3495243.3560519

Cited by 40 publications

(16 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, different from generative methods, there are studies on employing self-supervised learning techniques to exploit available unlabeled data directly to their advantage. Depending on the modalities involved in the model design and deployment phases, existing studies can be divided into the categories of single-modality [31,33] and multi-modality [10,14,23,26,42].…”

Section: Related Workmentioning

confidence: 99%

“…CO-COA [10] performs contrastive learning between features extracted from multisensor data for fusion. Cosmo [26] designs a multimodal feature fusion contrastive method to fully use multimodal synergies within heterogeneous multimodal data. These solutions focus on improving the performance of multimodal HAR for scenarios where multimodal data are available.…”

Section: Related Workmentioning

confidence: 99%

“…Thus, the proposed framework operates in a multito-unimodal mode as shown in Figure 2 (c). Although similar twostage training frameworks (Figure 2 (a) & (b)) are used in previous works [14,26,33] to address label scarcity issues for unimodal or multimodal HAR, MESEN significantly differs from them due to the specific application scenarios involving modality limitations during the HAR model design and the deployment phases.…”

Section: Related Workmentioning

confidence: 99%

“…The most related work is Cosmo [26], which is capable of handling unlabeled multimodal data for HAR. However, it cannot be directly applied to our target application scenarios as it is designed to fuse multimodal data for scenarios where multimodal data are available during the deployment phase of HAR applications.…”

Section: Problem Statementmentioning

confidence: 99%

“…In particular, the recent advances in multimodal encoding in general artificial intelligence tasks, such as ImageBind [13], have implied that exploiting multimodal data for building HAR models is promising. The prior studies [9,21,22,26] have shown the HAR performance improvements brought by multimodal data fusion.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

Xu,

Gu,

Tan

et al. 2023

Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

Human activity recognition (HAR) will be an essential function of various emerging applications. However, HAR typically encounters challenges related to modality limitations and label scarcity, leading to an application gap between current solutions and real-world requirements. In this work, we propose MESEN, a multimodalempowered unimodal sensing framework, to utilize unlabeled multimodal data available during the HAR model design phase for unimodal HAR enhancement during the deployment phase. From a study on the impact of supervised multimodal fusion on unimodal feature extraction, MESEN is designed to feature a multi-task mechanism during the multimodal-aided pre-training stage. With the proposed mechanism integrating cross-modal feature contrastive learning and multimodal pseudo-classification aligning, MESEN exploits unlabeled multimodal data to extract effective unimodal features for each modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a few labeled samples. Extensive experiments on eight public multimodal datasets demonstrate that MESEN achieves significant performance improvements over state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal data. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computing methodologies → Machine learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Problem Statementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

Xu,

Gu,

Tan

et al. 2023

Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

show abstract

Deep learning for computer vision based activity recognition and fall detection of the elderly: a systematic review

Gaya-Morey,

Manresa-Yee,

Buades-Rubio

2024

Appl Intell

View full text Add to dashboard Cite

As the proportion of elderly individuals in developed countries continues to rise globally, addressing their healthcare needs, particularly in preserving their autonomy, is of paramount concern. A growing body of research focuses on Ambient Assisted Living (AAL) systems, aimed at alleviating concerns related to the independent living of the elderly. This systematic review examines the literature pertaining to fall detection and Human Activity Recognition (HAR) for the elderly, two critical tasks for ensuring their safety when living alone. Specifically, this review emphasizes the utilization of Deep Learning (DL) approaches on computer vision data, reflecting current trends in the field. A comprehensive search yielded 2,616 works from five distinct sources, spanning the years 2019 to 2023 (inclusive). From this pool, 151 relevant works were selected for detailed analysis. The review scrutinizes the employed DL models, datasets, and hardware configurations, with particular emphasis on aspects such as privacy preservation and real-world deployment. The main contribution of this study lies in the synthesis of recent advancements in DL-based fall detection and HAR for the elderly, providing insights into the state-of-the-art techniques and identifying areas for further improvement. Given the increasing importance of AAL systems in enhancing the quality of life for the elderly, this review serves as a valuable resource for researchers, practitioners, and policymakers involved in developing and implementing such technologies. Graphical abstract

show abstract

SGSM: semi-generalist sensing model combining handcrafted and deep learning methods

Yang,

Zhou,

Liu

et al. 2024

Int. J. Mach. Learn. & Cyber.

View full text Add to dashboard Cite

Cosmo

Cited by 40 publications

References 44 publications

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

Deep learning for computer vision based activity recognition and fall detection of the elderly: a systematic review

SGSM: semi-generalist sensing model combining handcrafted and deep learning methods

Contact Info

Product

Resources

About