Asymmetric Dynamic Attunement of Speech and Gestures in the Construction of Children’s Understanding

Abstract

As children learn they use their speech to express words and their hands to gesture. This study investigates the interplay between real-time gestures and speech as children construct cognitive understanding during a hands-on science task. 12 children (M = 6, F = 6) from Kindergarten (n = 5) and first grade (n = 7) participated in this study. Each verbal utterance and gesture during the task were coded, on a complexity scale derived from dynamic skill theory. To explore the interplay between speech and gestures, we applied a cross recurrence quantification analysis (CRQA) to the two coupled time series of the skill levels of verbalizations and gestures. The analysis focused on (1) the temporal relation between gestures and speech, (2) the relative strength and direction of the interaction between gestures and speech, (3) the relative strength and direction between gestures and speech for different levels of understanding, and (4) relations between CRQA measures and other child characteristics. The results show that older and younger children differ in the (temporal) asymmetry in the gestures–speech interaction. For younger children, the balance leans more toward gestures leading speech in time, while the balance leans more toward speech leading gestures for older children. Secondly, at the group level, speech attracts gestures in a more dynamically stable fashion than vice versa, and this asymmetry in gestures and speech extends to lower and higher understanding levels. Yet, for older children, the mutual coupling between gestures and speech is more dynamically stable regarding the higher understanding levels. Gestures and speech are more synchronized in time as children are older. A higher score on schools’ language tests is related to speech attracting gestures more rigidly and more asymmetry between gestures and speech, only for the less difficult understanding levels. A higher score on math or past science tasks is related to less asymmetry between gestures and speech. The picture that emerges from our analyses suggests that the relation between gestures, speech and cognition is more complex than previously thought. We suggest that temporal differences and asymmetry in influence between gestures and speech arise from simultaneous coordination of synergies.

Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

“…Such spectral decomposition approaches help quantify and correlate gesture and speech activity that happens on the syllable, clause, and sentence time scales. Others have used approaches in dynamical systems research, whereby changes in temporal structures of categorical events or continuous time series (recurrence quantification analysis, or RQA; Wallot, 2017;Webber & Marwan, 2015) are the object of study as opposed to averaged metrics of speech or gesture activity (De Jonge-Hoekstra, Van der Steen, Van Geert, & Cox, 2016;Fusaroli & Tylén, 2016;Pouw, De Jonge-Hoekstra, & Dixon, 2018). These RQA approaches are especially potent for understanding global gesture patterns that reflect stability of recurrent behavior through a time-dependent analysis of local fluctuations in gesture activity.…”

“…This question can now be addressed quantitatively via gesture network analysis. Given that we have two datasets where participants spoke a different language, namely the American-English DAF dataset (Pouw & Dixon, 2018) and the German SAGA dataset (De Jonge-Hoekstra et al, 2016;Lücking et al, 2010), we could approach this question via network analyses, and specifically we might want to use dichotomized adjacency networks for this. Before we proceed, it is however important to note that these results are not conclusive in any way as the current datasets are different in a number of ways next to the difference in spoken language.…”

Combining Recurrence Analysis and Automatic Movement Extraction from Video Recordings to Study Behavioral Coupling in Face-to-Face Parent-Child Interactions

“…It has been recently proposed that the directionality of the asymmetry between vertical and horizontal lines in a recurrence plot can provide complementary information about the coupling of two subsystems (Cox et al, 2016 ). De Jonge-Hoekstra et al ( 2016 ) applied cross-recurrence to analyse gestures and speech data streams recorded in children during a hands-on science task and found that in 5-year-old children such asymmetry in the resulting recurrence plots suggests that speech categories attract gesture categories at the same level of abstraction in a more dynamically stable fashion than vice versa . They also showed that gestures and speech are more synchronized when children grow older.…”

Categorical Cross‐Recurrence Quantification Analysis Applied to Communicative Interaction during Ainsworth’s Strange Situation

“…Cross-Recurrence Quantification Analysis (C-RQA) is used with signals coming from two interacting systems [44][45][46][47][48][49][50]. C-RQA, like RQA, quantifies coordinative patterns based on an analysis of the sequence of behaviours performed in real time [40,41].…”

“…In psychology, such signals can be fluctuations in gait, postural changes, eye movements, or informational patterns, such as syntactic structures or words exchanged by two persons during a communicative interaction. In fact, there is a large corpus of evidence in which these nonlinear techniques have been used to analyse postural fluctuations [42], conversational interaction between caregiver and child [34,51,52], coupling of time series of verbalizations and gestures [50], and reading comprehension [53]. Additionally, Complexity 5 through these analyses, interpersonal coordination has been characterized in terms of coupling of eye movements [42], body movements [41,54], child behaviour and sleep [55], patterns of aggression [29], minimal and effective forms of coordination during the dialogue [4], and early language development [46,47].…”