Action-conditioned On-demand Motion Generation

Lu, Qiujing; Zhang, Yipeng; Lu, Mingjian; Roychowdhury, Vwani P.

doi:10.1145/3503161.3548287

Cited by 13 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multimodal learning In multimodal learning, different modalities, e.g., texts, images (Zhang et al 2023), videos (Zadeh et al 2016(Zadeh et al , 2018, Magnetic Resonance Image (MRI) modalities (Dinsdale et al 2021) or other sources of information (Lu et al 2022;Zhang et al 2022), are often considered to be complementary to each other. Fusion has been widely studied as one of the most critical topics in multimodal learning.…”

Section: Related Workmentioning

confidence: 99%

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Wang,

Luo,

et al. 2024

AAAI

View full text Add to dashboard Cite

Multimodal learning with incomplete input data (missing modality) is very practical and challenging. In this work, we conduct an in-depth analysis of this challenge and find that modality dominance has a significant negative impact on the model training, greatly degrading the missing modality performance. Motivated by Grad-CAM, we introduce a novel indicator, gradients, to monitor and reduce modality dominance which widely exists in the missing-modality scenario. In aid of this indicator, we present a novel Gradient-guided Modality Decoupling (GMD) method to decouple the dependency on dominating modalities. Specifically, GMD removes the conflicted gradient components from different modalities to achieve this decoupling, significantly improving the performance. In addition, to flexibly handle modal-incomplete data, we design a parameter-efficient Dynamic Sharing (DS) framework which can adaptively switch on/off the network parameters based on whether one modality is available. We conduct extensive experiments on three popular multimodal benchmarks, including BraTS 2018 for medical segmentation, CMU-MOSI, and CMU-MOSEI for sentiment analysis. The results show that our method can significantly outperform the competitors, showing the effectiveness of the proposed solutions. Our code is released here: https://github.com/HaoWang420/Gradient-guided-Modality-Decoupling.

show abstract

Section: Related Workmentioning

confidence: 99%

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Wang,

Luo,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Given that EEG samples were of varying lengths, we adopted a novel stratified training approach which is widely used in the processing time-series data in the multimedia community. 13 In each training iteration, for each patient, we randomly selected starting time stamps and constructed FFT maps from 15 second windows, such that information contributed from each patient was equal in each training iteration, with positive and negative samples balanced during the training phase. We conducted 400 training iterations in each fold.…”

Section: Model Training and Evaluationmentioning

confidence: 99%

Application of an EEG-based deep learning model to discriminate children with epileptic spasms from normal controls

Zhang

Daida

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Objective: Given that epileptic spasms are often subtle, and that identification of hypsarrhythmia is limited by inadequate inter-rater reliability, there is a significant need for novel tools to aid the clinical identification of Infantile Epileptic Spasms Syndrome (IESS). Deep learning is an emerging technology which may enable efficient classification of disease states and may facilitate discovery of novel biomarkers. In this study, we set out to evaluate whether children with epileptic spasms can be distinguished from normal controls with use of an EEG-based deep learning model. Methods: A deep learning model was trained and validated (5-fold cross validation) using 400 EEG samples (2 awake and 2 sleep samples from 50 children with epileptic spasms and 50 normal controls). Salient frequency bands and specific morphologic EEG features were identified with occlusion sensitivity analysis and targeted input perturbation, respectively. Results: The model accurately distinguishes children with epileptic spasms from normal controls, solely on the basis of relatively short EEG samples. Using sleep data, accuracy = 0.95, recall = 0.96, precision (sensitivity) = 0.94, specificity = 0.94, and F1 score = 0.95. With awake data, accuracy = 0.91, recall = 0.84, precision = 0.98, specificity = 0.98, and F1 score = 0.90. The salient frequency bands for classification are 9.7 - 22.0 Hz and 1.0 - 6.8 Hz in sleep and awake EEG, respectively. With visual analysis of extracted salient features, we suspect that the model is identifying cases on the basis of paroxysmal fast activity in sleep and spike-wave activity in wakefulness. Conclusion: This deep learning model represents a first step in the development of efficient algorithms that may aid in identification of epileptic spasms and IESS. More importantly, this approach may facilitate novel EEG-based biomarkers of epileptic spasms.

show abstract

“…We evaluate the diversity of motions produced by the low-level model. Following [Hassan et al 2021;Lu et al 2022;…”

Section: Low-level Conditional Policymentioning

confidence: 99%

C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters

Dou,

Chen,

Fan

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

Figure 1: Our framework enables physically simulated characters to master highly varied and extensive skills with high efficiency and effectiveness. Notably, it offers an explicit control handle for directly specifying the desired skill from a diverse and extensive set of skills. Here, a character is instructed by the user to perform a sequence of skills, including kick, jump attack, sword bash, shield bash, and finally, roaring.

show abstract

Action-conditioned On-demand Motion Generation

Cited by 13 publications

References 32 publications

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Application of an EEG-based deep learning model to discriminate children with epileptic spasms from normal controls

C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters

Contact Info

Product

Resources

About