CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Abstract

Over the past few years, the field of visual social cognition and face processing has been 1 dramatically impacted by a series of data-driven studies employing computer-graphics 2 tools to synthesize arbitrary meaningful facial expressions. In the auditory modality, 3 reverse correlation is traditionally used to characterize sensory processing at the level of 4 spectral or spectro-temporal stimulus properties, but not higher-level cognitive 5 processing of e.g. words, sentences or music, by lack of tools able to manipulate the 6 stimulus dimensions that are relevant for these processes. Here, we present an 7 open-source audio-transformation toolbox, called CLEESE, able to systematically 8 randomize the prosody/melody of existing speech and music recordings. CLEESE works 9 by cutting recordings in small successive time segments (e.g. every successive 100 10 milliseconds in a spoken utterance), and applying a random parametric transformation 11 of each segment's pitch, duration or amplitude, using a new Python-language 12 implementation of the phase-vocoder digital audio technique. We present here two 13 applications of the tool to generate stimuli for studying intonation processing of 14 interrogative vs declarative speech, and rhythm processing of sung melodies.15 20 facial metrics such as width-to-height ratio, acoustical features such as mean pitch) are 21 posited by the experimenter before being controlled or tested experimentally, which may 22 create a variety of confirmation biases or experimental demands. For instance, stimuli 23 constructed to display western facial expressions of happiness or sadness may well be 24 recognized as such by non-western observers [1], but yet may not be the way these 25 emotions are spontaneously produced, or internally represented, in such cultures [2]. 26Similarly in auditory cognition, musical stimuli recorded by experts pressed to express 27 emotions in music may do so by mimicking expressive cues used in speech, but these 28 cues may not exhaust the many other ways in which arbitrary music can express 29 emotions [3]. For all these reasons, in recent years, a series of powerful data-driven 30 PLOS1/17 33 systematically-varied stimuli [5]. 34 The reverse correlation technique was first introduced in neurophysiology to 35 characterize neuronal receptive fields of biological systems with so-called "white noise 36 analysis" [6-9]. In psychophysics, the technique was then adapted to characterize 37 human sensory processes, taking behavioral choices (e.g., yes/no responses) instead of 38 neuronal spikes as the systems' output variables to study, e.g. in the auditory domain, 39 detection of tones in noise [10] or loudness weighting in tones and noise ( [11]; see [4] for 40 a review of similar applications in vision). In the visual domain, these techniques have 41 been extended in recent years to address not only low-level sensory processes, but 42 higher-level cognitive mechanisms in humans: facial recognition [12], emotional 43 expressions [2, 13], social traits [14], as well a...

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

“…Changes in the subglottal pressure due to the contraction of thoracic and abdominal muscles, which are controlled from the anterior horn of the spinal cord, primarily lead to modulations of voice intensity. At moderate intensities, such as in happy, aroused voices compared to calm or sad voices, the effect of the modulation is carried linearly through the vocal pathway and can be simulated with a simple scalar multiplication of the recording’s root mean square (RMS) intensity (see e.g., Ilie & Thompson, 2006) or, for arbitrary intensity profiles, a piece-wise linear function as implemented for example in the reverse-correlation toolbox CLEESE 8 (Burred, Ponsot, Goupil, Liuni, & Aucouturier, 2019).…”

“…Simple algorithms, as used for example in altered auditory feedback research (Hain, Burnett, Larson, & Kiran, 2001) and implemented in the DAVID toolbox 10 (Rachman et al, 2018), are based on resampling or multiple delay lines (a technique that introduces a small delay to an audio signal in order to play it faster/slower, thus raising/lowering its pitch; Dattorro, 1997) and may alter vocal tract filtering or formants unrealistically beyond small parametric changes. State-of-the-art techniques that allow separating source and filter information to avoid such artefacts are based on reconstructions of the signal’s short-time Fourier transform (STFT) at nonuniform rates, such as the pitch synchronous overlap and add (PSOLA) method as implemented for example in PRAAT (Boersma & Weenink, 2002); the phase-vocoder method (Moulines & Laroche, 1995) as implemented for example in CLEESE (Burred et al, 2019); or pitch-adaptive analyses techniques such as the adaptive interpolation of weighted spectrum method as implemented in STRAIGHT 11 (Kawahara, 1997). These transformation methods not only allow raising or lowering the mean pitch of a recording, which may correspond to a baseline change of valence (see Figure 3A; see e.g., Ilie & Thompson, 2006), but can also manipulate the difference between the instantaneous and mean F0 to exaggerate or lessen variations, as seen for example in fearful versus sad vocalizations (see Figure 3B; see e.g., Pell & Kotz, 2011); create parametric F0 contours such as vibrato in anxious voices (Figure 3C; see e.g., Bachorowski & Owren, 1995), or local intonations at the start or end of an utterance, as in surprised or assertive speech (see Figure 3D; see e.g., Jiang & Pell, 2017).…”

“…Sound Example S4: Female speech, first: original; second–sixth: random variations of speed rate, generated on six successive time-windows, using Gaussian distributions centered at ±0%, SD = ±10% original duration. All transformations made with a phase-vocoder, using the CLEESE toolbox (Burred et al, 2019).…”

“…Sound Example S5: Female speech, first: original; second–sixth: random variations of vocal tract filter, generated with spectral envelope frequency warping using the CLEESE toolbox (Burred et al, 2019).…”

“…While vocal tract characteristics have been mainly studied in the context of speaker identity(Mohammadi and Kain 2017b), recent work has suggested they are also actively manipulated by emotional oro-facial gestures such as those involved in the expression of disgust(Chong et al 2018) or smiling(Arias et al 2018a). Sound example S5: Female speech, first: original; second-sixth: random variations of vocal tract filter, generated with spectral envelope frequency warping using the CLEESE toolbox(Burred et al 2019).Sound example S6: Female speech, first & third: original; second & fourth:formant shifted upwards to simulate smiling, generated with frequency warping(Arias et al 2018b).…”