Improving the Movement Synchrony Estimation with Action Quality Assessment in Children Play Therapy

Abstract

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The automatic measurement of movement synchrony is worth studying for social behavior analysis applications, for instance, in play therapy of children in the autism spectrum. Existing approaches based on motion energy analysis are strongly reliant on the region of interest, and thus limit the interaction between individuals, especially for highly engaging activities like play therapy. Inspired by action quality assessment, a task to assess how well an action has been performed, in this paper, we propose an end-to-end deep learning method to integrate the following major tasks: (1) the automatic assessment of children's performance in play therapy, and (2) the automatic estimation of movement synchrony between children and therapists, facilitated by an auxiliary task of intervention activity recognition. This multi-task paradigm generally improves the performance of our model across all tasks. Furthermore, when annotations are subjective, the typical exclusive annotation strategy may reduce tagging quality. As a result, we explored applying distribution learning to mitigate human bias in movement synchrony estimation. We allowed the second and third labels for each instance, namely the uncertainty-preserved annotation approach. We tested our method on Play Therapy 13 (PT13), a dataset collected from video recordings of play therapy interventions. The findings of the experiments indicated that our framework can accurately quantify movement synchronization and assess the quality of children's actions in play therapy. Moreover, the uncertainty-preserved annotation approach produced a comparable outcome to standard methods at a far reduced cost, demonstrating its efficacy in mitigating biases.

Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment

“…Our work can be classified under AQA and SA, which involves the computer vision-based quantification of the quality of movements and actions. Works in AQA and SA have mainly been focused on domains like physiotherapy [6,19,25,33,36], Olympic sports [3,24,28,35,39,41], various types of skills [5,20,26,38]. However, workout form assessment, especially, in real-world conditions, has not received much attention.…”

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

“…Calabrò et al [29] reconstructed the Inter Beat Intervals (IBI) segments of electrocardiogram data via a convolutional auto-encoder to explore dyadic interaction between children and therapists. Li et al [5] proposed a multi-task framework to integrate movement synchrony estimation with auxiliary tasks such as intervention activity recognition and individual action quality assessment. Nevertheless, both works [5], [29] necessitate access to original video recordings, making them non-privacy-preserving.…”

“…Li et al [5] proposed a multi-task framework to integrate movement synchrony estimation with auxiliary tasks such as intervention activity recognition and individual action quality assessment. Nevertheless, both works [5], [29] necessitate access to original video recordings, making them non-privacy-preserving.…”

“…We evaluated our framework on two datasets: Play Therapy 13 dataset (PT13) [5] and Two-person Action Synchronized Diving dataset (TASD-2) [6].…”

“…To justify our work, we performed experiments on two datasets: (1) PT13 dataset [5], a real-world dataset on autism therapy, and (2) TASD-2 dataset [6] collected from synchronized diving competitions. Our method surpassed its competitors for movement synchrony estimation without mandating access to the original videos, which contain sensitive personal information.…”