Cracking the social code of speech prosody using reverse correlation

Ponsot, Emmanuel; Burred, Juan José; Belin, Pascal; Aucouturier, Jean‐Julien

doi:10.1073/pnas.1716090115

Cited by 81 publications

(83 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SIR can also be applied to study any parametrizable sensory stimulus spaces (e.g. auditory 20,21,53,54 , as well as other cognitive, social and affective tasks (for reviews see 19,55,55,56 , and to study the information processing mechanisms of both brain and in silicon architectures.…”

Section: Resultsmentioning

confidence: 99%

“…Stimulus information can take different forms. It can consist of images generated by the random selection of pixels [13][14][15][16] , or by sampling from generative models of complex stimuli [17][18][19][20][21] . Such randomly generated images (as described in more detail below) are then presented to participants, who are asked to categorize them.…”

Section: The Sir Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Making thebrain-activity-to-informationleap using a novel framework: Stimulus Information Representation (SIR)

Schyns

Ince

2019

Preprint

View full text Add to dashboard Cite

It is no good poking around in the brain without some idea of what one is looking for. That would be like trying to find a needle in a haystack without having any idea what needles look like. The theorist is the [person] who might reasonably be asked for [their] opinion about the appearance of needles." HC Longuet-Higgins, 1969. AbstractA fundamental challenge in neuroscience is to understand how the brain processes information. Neuroscientists have approached this question partly by measuring brain activity in space, time and at different levels of granularity. However, our aim is not to discover brain activity per se, but to understand the processing of information that this activity reflects. To make this brain-activityto-information leap, we believe that we should reconsider brain imaging from the methodological foundations of psychology. With this goal in mind, we have developed a new data-driven framework, called Stimulus Information Representation (SIR), that enables us to better understand how the brain processes information from measures of brain activity and behavioral responses. In this article, we explain this approach, its strengths and limitations, and how it can be applied to understand how the brain processes information to perform behavior in a task.Much of human cognition starts with a brain that categorizes stimulus information to behave adaptively 1-3 . Thus, human brains are compulsive categorizers that use stimulus information to perform different categorization tasks. Consider the street scene shown on the left-hand side of Figure 1. The brain can perform numerous categorization tasks on this image: it can identify the country, city, and street; the houses and shops; the moving and stationary cars, their makes and models, age and condition; the people shopping or those just passing by, as well as their gait, identity, emotion and social interactions; it can also infer the weather, time of day, season, and so on.So, when we record brain activity, we need to circumscribe an experimental task in order to know which task the brain has performed when we analyze its activity, even when a task involves just a single image. And when we circumscribe a task, we still need to characterize the stimulus information that supports a particular categorization. Otherwise, we will not know what information the brain has processed when it categorized the street, or the make of a car, or the identity of a face or its expression, from the same image.Such task-relevant information processing is a generic, but often neglected theoretical point, that applies both to the interpretation of any sensory categorization in the brain and to its models. For example, Convolutional Neural Networks 4,5 (CNNs) are braininspired, hierarchically organized network models that also use stimulus information to perform multiple categorization tasks, with apparent human-like capabilities. But to realize the promise of CNNs as models of brain information processing 6-12 , a neglected pre-condition needs to be met -that these models ...

show abstract

Section: Resultsmentioning

confidence: 99%

Section: The Sir Frameworkmentioning

confidence: 99%

Making thebrain-activity-to-informationleap using a novel framework: Stimulus Information Representation (SIR)

Schyns

Ince

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The present work represents a conceptually-similar development for the auditory modality. paradigms (built on techniques such as reverse-correlation, classification image or 31 bubbles; see [4] for a review) were introduced in the field of visual cognition to discover 32 relevant signal features empirically, by analyzing participant responses to large sets of 48 All of these techniques aim to isolate the subspace of feature dimensions that 49 maximizes participant responses, and as such, need to search the stimulus generative 50 space e.g. of all possible images or sounds relevant for a given task.…”

Section: Introduction 16mentioning

confidence: 99%

“…285 We give here a proof of concept of how to use CLEESE in a reverse-correlation 286 experiment to uncover what exact pitch contour drives participants' categorization of an 287 utterance as interrogative or declarative. Data come from the first experiment presented 288 in [31].…”

mentioning

confidence: 99%

“…Because 430 this correlation may also be driven by the amount of attention that participants had 431 both in the PROMS and in the reverse-correlation tasks (which may be similar), further 432 work is required to determine the exact nature of this observed relationship. underlie judgements of speaker dominance and trustworthiness in short utterances like 447 the word 'hello' [31]. The technique allowed to establish that both constructs 448 corresponded to robust and distinguishing pitch trajectories, which remained 449 remarkably stable whether males or female listeners judged male or female speakers.…”

mentioning

confidence: 99%

See 1 more Smart Citation

CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Burred¹,

Ponsot²,

Goupil³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Over the past few years, the field of visual social cognition and face processing has been 1 dramatically impacted by a series of data-driven studies employing computer-graphics 2 tools to synthesize arbitrary meaningful facial expressions. In the auditory modality, 3 reverse correlation is traditionally used to characterize sensory processing at the level of 4 spectral or spectro-temporal stimulus properties, but not higher-level cognitive 5 processing of e.g. words, sentences or music, by lack of tools able to manipulate the 6 stimulus dimensions that are relevant for these processes. Here, we present an 7 open-source audio-transformation toolbox, called CLEESE, able to systematically 8 randomize the prosody/melody of existing speech and music recordings. CLEESE works 9 by cutting recordings in small successive time segments (e.g. every successive 100 10 milliseconds in a spoken utterance), and applying a random parametric transformation 11 of each segment's pitch, duration or amplitude, using a new Python-language 12 implementation of the phase-vocoder digital audio technique. We present here two 13 applications of the tool to generate stimuli for studying intonation processing of 14 interrogative vs declarative speech, and rhythm processing of sung melodies.15 20 facial metrics such as width-to-height ratio, acoustical features such as mean pitch) are 21 posited by the experimenter before being controlled or tested experimentally, which may 22 create a variety of confirmation biases or experimental demands. For instance, stimuli 23 constructed to display western facial expressions of happiness or sadness may well be 24 recognized as such by non-western observers [1], but yet may not be the way these 25 emotions are spontaneously produced, or internally represented, in such cultures [2]. 26Similarly in auditory cognition, musical stimuli recorded by experts pressed to express 27 emotions in music may do so by mimicking expressive cues used in speech, but these 28 cues may not exhaust the many other ways in which arbitrary music can express 29 emotions [3]. For all these reasons, in recent years, a series of powerful data-driven 30 PLOS1/17 33 systematically-varied stimuli [5]. 34 The reverse correlation technique was first introduced in neurophysiology to 35 characterize neuronal receptive fields of biological systems with so-called "white noise 36 analysis" [6-9]. In psychophysics, the technique was then adapted to characterize 37 human sensory processes, taking behavioral choices (e.g., yes/no responses) instead of 38 neuronal spikes as the systems' output variables to study, e.g. in the auditory domain, 39 detection of tones in noise [10] or loudness weighting in tones and noise ( [11]; see [4] for 40 a review of similar applications in vision). In the visual domain, these techniques have 41 been extended in recent years to address not only low-level sensory processes, but 42 higher-level cognitive mechanisms in humans: facial recognition [12], emotional 43 expressions [2, 13], social traits [14], as well a...

show abstract

Psychology

Imhof¹

2020

The Handbook of Listening

View full text Add to dashboard Cite

Cracking the social code of speech prosody using reverse correlation

Cited by 81 publications

References 45 publications

Making thebrain-activity-to-informationleap using a novel framework: Stimulus Information Representation (SIR)

Making thebrain-activity-to-informationleap using a novel framework: Stimulus Information Representation (SIR)

CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Psychology

Contact Info

Product

Resources

About