Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

Song, Meishu; Yang, Zijiang; Triantafyllopoulos, Andreas; Nakamura, Toru; Zhang, Yongxin; Ren, Zhao; Takeuchi, Hiroki; Kishi, Akifumi; Ishizawa, Tetsuro; Yoshiuchi, Kazuhiro; Zhang, Haojie; Qian, Kun; Hu, Bin; Schuller, Björn W.; Yamamoto, Yoshiharu

doi:10.1109/icdmw60847.2023.00167

Cited by 2 publications

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study

Song,

Yang,

Triantafyllopoulos

et al. 2024

JMIR Ment Health

View full text Add to dashboard Cite

Background:The field of mental health technology presently has significant gaps that need addressing, particularly in the domain of daily monitoring and personalized assessments. Current non-invasive devices like wristbands and smartphones are capable of collecting a wide range of data, which has not yet been fully utilized for mental health monitoring. Objective:The paper aims to introduce a novel dataset for Personalized Daily Mental Health Monitoring and a new Macro-Micro Framework. This framework is designed to employ multimodal and multitask learning strategies for improved personalization and prediction of emotional states in individuals.Methods: Data was collected from 242 individuals using wristbands and smartphones, capturing physiological signals, speech data, and self-annotated emotional states. The proposed framework combines macro-level emotion transformer embeddings with micro-level personalization layers specific to each user. It also introduces a dynamic restrained uncertainty weighting method to effectively integrate various data types for a balanced representation of emotional states. Several fusion techniques, personalization strategies, and multitask learning approaches were explored. Results:The proposed framework was evaluated using the Concordance Correlation Coefficient (CCC), resulting in a score of 0.503. This result demonstrates the framework's efficacy in predicting emotional states. Conclusions:The paper concludes that the proposed multimodal and multitask learning framework, which leverages transformerbased techniques and dynamic task weighting strategies, is superior for the personalized monitoring of mental health. The study indicates the potential of transforming daily mental health monitoring into a more personalized application, opening up new avenues for technology-based mental health interventions.

show abstract

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study

Song,

Yang,

Triantafyllopoulos

et al. 2024

JMIR Ment Health

View full text Add to dashboard Cite

show abstract

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study (Preprint)

Song,

Yang,

Triantafyllopoulos

et al. 2024

Preprint

View full text Add to dashboard Cite

BACKGROUND The field of mental health technology presently has significant gaps that need addressing, particularly in the domain of daily monitoring and personalized assessments. Current non-invasive devices like wristbands and smartphones are capable of collecting a wide range of data, which has not yet been fully utilized for mental health monitoring. OBJECTIVE The paper aims to introduce a novel dataset for Personalized Daily Mental Health Monitoring and a new Macro-Micro Framework. This framework is designed to employ multimodal and multitask learning strategies for improved personalization and prediction of emotional states in individuals. METHODS Data was collected from 242 individuals using wristbands and smartphones, capturing physiological signals, speech data, and self-annotated emotional states. The proposed framework combines macro-level emotion transformer embeddings with micro-level personalization layers specific to each user. It also introduces a dynamic restrained uncertainty weighting method to effectively integrate various data types for a balanced representation of emotional states. Several fusion techniques, personalization strategies, and multitask learning approaches were explored. RESULTS The proposed framework was evaluated using the Concordance Correlation Coefficient (CCC), resulting in a score of 0.503. This result demonstrates the framework's efficacy in predicting emotional states. CONCLUSIONS The paper concludes that the proposed multimodal and multitask learning framework, which leverages transformer-based techniques and dynamic task weighting strategies, is superior for the personalized monitoring of mental health. The study indicates the potential of transforming daily mental health monitoring into a more personalized application, opening up new avenues for technology-based mental health interventions.

show abstract

Crossmodal Transformer on Multi-Physical Signals for Personalised Daily Mental Health Prediction

Cited by 2 publications

References 26 publications

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study

Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study (Preprint)

Contact Info

Product

Resources

About