An Asymmetric Modeling for Action Assessment

Gao, Jibin; Zheng, Wei‐Shi; Pan, Jiahui; Gao, Chengying; Wang, Yaowei; Zeng, Wei; Lai, Jianhuang

doi:10.1007/978-3-030-58577-8_14

Cited by 43 publications

(21 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AQA is a task that assesses how well a certain action is performed [60]. Sports video analysis [43,44,46,60,65] accounts for a large portion for AQA, and it also includes topics like health care [22,31], instructional video analysis [18,19] and arts [45]. Recently graph-based methods [41,66] have exhibited superior performance in global (whole-body) and local (joints coordination) balancing and interpretability.…”

Section: Action Quality Assessmentmentioning

confidence: 99%

“…The AQA, which is also known as skill determination, is a task to assess how well an action has been performed [60]. Although AQA has been widely applied to a variety of areas such as sports [22,41,44,60], arts [45], medical [22,31] and education [19], to the best of our knowledge, not much effort has been put into autism treatment. For the framework evaluation, we prepared and used our in-house dataset that includes video recordings of play therapy with children participants.…”

Section: Introductionmentioning

confidence: 99%

“…Our framework takes a pair of frame sequences, namely the "child" and "therapist, " from the same video recording of play therapy, and predicts the movement synchrony level between them while assessing the child's actions based on how well he/she imitates and follows the therapists. This process is analogous to the synchronized diving assessment, wherein AQA and MS module evaluate execution and synchronization of the action, respectively [22]. The framework also predicts the activity classes of the intervention as an auxiliary task to boost overall performance.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving the Movement Synchrony Estimation with Action Quality Assessment in Children Play Therapy

Bhat

Barmaki

2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The automatic measurement of movement synchrony is worth studying for social behavior analysis applications, for instance, in play therapy of children in the autism spectrum. Existing approaches based on motion energy analysis are strongly reliant on the region of interest, and thus limit the interaction between individuals, especially for highly engaging activities like play therapy. Inspired by action quality assessment, a task to assess how well an action has been performed, in this paper, we propose an end-to-end deep learning method to integrate the following major tasks: (1) the automatic assessment of children's performance in play therapy, and (2) the automatic estimation of movement synchrony between children and therapists, facilitated by an auxiliary task of intervention activity recognition. This multi-task paradigm generally improves the performance of our model across all tasks. Furthermore, when annotations are subjective, the typical exclusive annotation strategy may reduce tagging quality. As a result, we explored applying distribution learning to mitigate human bias in movement synchrony estimation. We allowed the second and third labels for each instance, namely the uncertainty-preserved annotation approach. We tested our method on Play Therapy 13 (PT13), a dataset collected from video recordings of play therapy interventions. The findings of the experiments indicated that our framework can accurately quantify movement synchronization and assess the quality of children's actions in play therapy. Moreover, the uncertainty-preserved annotation approach produced a comparable outcome to standard methods at a far reduced cost, demonstrating its efficacy in mitigating biases.

show abstract

Section: Action Quality Assessmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving the Movement Synchrony Estimation with Action Quality Assessment in Children Play Therapy

Bhat

Barmaki

2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…Human-centric performance assessment, which aims to predict the performance score of human individuals in certain kinds of activities, is attracting increasing attention due to its broad applications in multiple areas. Some researchers focus on assessing the performance of athletes in Olympic events, such as diving, figure skating and gymnastic vault [39,42,43,46,49,60], some target on skill determination in daily activities [15,16], others address medical issues like surgical skill [20,21,67] and patient rehabilitation [41,50]. In comparison, little attention is paid to entertainment videos like viral dance clips.…”

Section: Related Workmentioning

confidence: 99%

“…A segment-based pseudo-3D residual network (P3D) [47] is proposed in [60] to extract features at segment-level for better performance. Later works like [20,49] employ I3D [9] model for segmental feature extraction. In [49], an uncertainty-aware score distribution learning (USDL) approach is proposed to handle the score ambiguity caused by multiple judges in sports events.…”

Section: Related Workmentioning

confidence: 99%

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Wang

Weng

et al. 2022

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Dance challenges are going viral in video communities like TikTok nowadays. Once a challenge becomes popular, thousands of short-form videos will be uploaded within a couple of days. Therefore, virality prediction from dance challenges is of great commercial value and has a wide range of applications, such as smart recommendation and popularity promotion. In this article, a novel multi-modal framework that integrates skeletal, holistic appearance, facial and scenic cues is proposed for comprehensive dance virality prediction. To model body movements, we propose a pyramidal skeleton graph convolutional network (PSGCN) that hierarchically refines spatio-temporal skeleton graphs. Meanwhile, we introduce a relational temporal convolutional network (RTCN) to exploit appearance dynamics with non-local temporal relations. An attentive fusion approach is finally proposed to adaptively aggregate predictions from different modalities. To validate our method, we introduce a large-scale viral dance video (VDV) dataset, which contains over 4,000 dance clips of eight viral dance challenges. Extensive experiments on the VDV dataset well demonstrate the effectiveness of our approach. Furthermore, we show that short video applications such as multi-dimensional recommendation and action feedback can be derived from our model.

show abstract

Surgical Skill Assessment via Video Semantic Aggregation

Wang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Asymmetric Modeling for Action Assessment

Cited by 43 publications

References 28 publications

Improving the Movement Synchrony Estimation with Action Quality Assessment in Children Play Therapy

Improving the Movement Synchrony Estimation with Action Quality Assessment in Children Play Therapy

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Surgical Skill Assessment via Video Semantic Aggregation

Contact Info

Product

Resources

About