Does our speech change when we cannot gesture?

Hoetjes, Marieke; Krahmer, Emiel; Swerts, Marc

doi:10.1016/j.specom.2013.06.007

Cited by 25 publications

(41 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We add this condition as it has been shown that anticipatory postural adjustments (APA's) that arise when moving the upper limbs while standing are dramatically diminished when the body is in a more stable sitting position (Cordo & Nasher, 1982). This could also explain that Krahmer & Swerts (2006) did not find effects of body movements on F0 and intensity measures as their participants were sitting throughout the experiments (also see Hoetjes, Krahmer, & Swerts, 2013). Thus if APA's are driving the current effects on phonation then we would find that the effects of upper limb movements on phonation are absent or diminished in the sitting condition relative to upper limb movement effects on phonation in the standing condition.…”

Section: Pre-registration Confirmatory Studymentioning

confidence: 73%

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

Wtjl¹,

Sa²,

Ja³

2018

Preprint

View full text Add to dashboard Cite

Hand gestures during speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity) and speech (e.g., peaks in Fundamental Frequency). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners' understanding. However, gesturespeech synchrony has been invariably understood as a "neural-cognitive" achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium -the body. The current paper presents an exploratory study together with a pre-registration of a larger scale confirmatory study. We provide preliminary evidence that upper limb motions with greater momentum (i.e., physical impetus) affects Fundamental Frequency and the Amplitude Envelope of phonation in a way that accommodates research on gesture-speech synchrony. We speculate that anticipatory postural adjustments and related physical effects of upper limb movements on the musculoskeletal connective system could lead to changes in alveolar (lung) pressure that can impart (prosodic) contrasts in phonation. Here we pre-registered a confirmatory study to more comprehensively address this hypothesis.

show abstract

Section: Pre-registration Confirmatory Studymentioning

confidence: 73%

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

Wtjl¹,

Sa²,

Ja³

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…In line with our previous study on repeated references (Hoetjes et al, 2015), and following many other studies (e.g., Alibali et al, 2001;Bard et al, 2000;Bavelas et al, 2008;de Ruiter et al, 2012;Hoetjes et al, 2014;Mol et al, 2009), we include visibility as an additional variable in the design of our production experiment (experiment I). We do this in such a way that one group of participants will be able to see each other (mutual visibility), while the other group are prevented to do so using a screen (no visibility).…”

Section: On the Role Of Visibilitymentioning

confidence: 99%

“…As mentioned above, following Hoetjes et al (2015), and many other related studies (e.g., Alibali et al, 2001;Bavelas et al, 2008;de Ruiter et al, 2012;Hoetjes et al, 2014;Mol et al, 2009), we added visibility as an additional variable to the design, in such a way that one group of participants could see each other during the experiment, while the other group was prevented from doing so by an opaque screen which was placed in between them.…”

Section: The Present Studymentioning

confidence: 99%

On what happens in gesture when communication is unsuccessful

Hoetjes

Krahmer

Swerts

2015

Speech Communication

Self Cite

View full text Add to dashboard Cite

On what happens in gesture when communication is unsuccessfulHoetjes, Marieke; Krahmer, Emiel; Swerts, Marc General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.-Users may download and print one copy of any publication from the public portal for the purpose of private study or research -You may not further distribute the material or use it for any profit-making activity or commercial gain -You may freely distribute the URL identifying the publication in the public portal Take down policyIf you believe that this document breaches copyright, please contact us providing details, and we will remove access to the work immediately and investigate your claim. AbstractPrevious studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director-matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback.

show abstract

“…Thus evidence is still lacking for theories that propose a pivotal role for gesture in speech-production processes (e.g., Krauss, 1998;McNeill, 1992McNeill, , 2005. Indeed, Hoetjes, Krahmer, & Swerts (2014) conclude that there is little convincing evidence for direct (beneficial) effects of gesture on speech properties (e.g., fluency, acoustic parameters), especially in the case of spontaneous gestures in natural contexts.…”

Section: Introductionmentioning

confidence: 96%

“…Pouw, van Gog, Zwaan, Paas, 2017), so as to successfully summon concepts that the speech production system fails to achieve unimodally (i.e., semantic fluency). As such, it has been theorized that primarily semantics are in gesture's functional domain (e.g., Hoetjes et al, 2014;Hosetter & Alibali, 2008). Of course, there has been research on non-semantic aspects of the gesture-speech relation.…”

Section: Introductionmentioning

confidence: 99%

Stabilizing Speech Production through Gesture-Speech Coordination

Wtjl¹,

Jonge-Hoekstra²,

Ja³

2018

Preprint

View full text Add to dashboard Cite

Hand-gestures are seamlessly coordinated with speech. Yet, there is only anecdotal support for gestures’ functional role in speech production. Here we explore temporal aspects of speech production when people use hand gesture. We performed exploratory analyses with a naturalistic German-speaking sample from The Bielefeld Speech and Gesture Alignment Corpus (SaGA), which consisted of 67 minutes of narration data and over 500 gesture events (N = 6). We found that the rhythmic timing of speech (defined as the mean and standard deviations of speech onset intervals) is highly correlated with the likelihood of gesturing. Furthermore, we utilized deep learning methods to track gesture motion, and extracted the amplitude envelope of speech, so as to gauge the degree of (continuous) gesture-speech synchrony. We then performed a continuous time-series analysis (recurrence quantification analysis; RQA) to index how temporal properties of speech change when gesture and speech are more or less synchronized. Our analyses revealed that when gesture and speech were more synchronized, the temporal structure of speech was more ordered and less complex, as indexed by classic measures of dynamic temporal stability (e.g., Entropy, Ratio of %Determinism/Recurrence). We suggest that a fundamental gesture-speech relation is rooted in entrainment, which yields stability in the temporal structure of speech.

show abstract

Does our speech change when we cannot gesture?

Cited by 25 publications

References 32 publications

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

On what happens in gesture when communication is unsuccessful

Stabilizing Speech Production through Gesture-Speech Coordination

Contact Info

Product

Resources

About