Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

Abstract

While acoustic analysis methods have become a commodity in voice emotion research, experiments that attempt not only to describe but to computationally manipulate expressive cues in emotional voice and speech have remained relatively rare. We give here a nontechnical overview of voice-transformation techniques from the audio signal-processing community that we believe are ripe for adoption in this context. We provide sound examples of what they can achieve, examples of experimental questions for which they can be used, and links to open-source implementations. We point at a number of methodological properties of these algorithms, such as being specific, parametric, exhaustive, and real-time, and describe the new possibilities that these open for the experimental study of the emotional voice.

Paralinguistic Features Communicated through Voice can Affect Appraisals of Confidence and Evaluative Judgments

“…Thus, this two-way interaction between thought direction and pitch was interpreted as a case in which speaker confidence (arising from low pitch) validated thoughts because thought confidence but not thought favorability mediated the effect. This research is consistent with recent research showing that pitch and other voice features can be manipulated to study their impact on the speaker (Arias et al, 2021 ). Further examples of research that illustrate the effect of other features of the speaker affecting attitudes via meta-cognitive validation of thoughts can be found in Briñol and Petty ( 2009 ).…”

Comment: Advances in Studying the Vocal Expression of Emotion: Current Contributions and Further Options

“…Issues such as the vocal mechanisms used in the emotional interpretation of the composer’s score by the singer (and the effect of the singer’s emotional state during the performance), as well as the acoustic cues used by the listener to infer emotional content and the differential effect on the resulting aesthetic emotions (see Coutinho et al, 2019) still remain to be further investigated. The study of emotion inference from vocal variations in singing could very profitably be conducted using the advanced techniques proposed by Arias et al (2021) and Schuller and Schuller (2021) in this issue. In his authoritative survey of the brain mechanisms underlying prosody, Grandjean (2021) also mentions music.…”

The shallow of your smile: the ethics of expressive vocal deep-fakes

“…Recent progress in signal processing have indeed made possible the real-time manipulation of e.g. facial expressions such as smiles [2] and vocal expressive cues such as pitch [56] or timbre [2]. Perhaps even more radically, recent advances in deep neural network [17]); (B) Manipulation of individual action units in still photographs using Generative Adversarial Networks (GANimation [55]); (C) Real-time smile filters in commercial video sharing plateforms (Tiktok, ByteDance Ltd., Beijing, China); (D) Still from the Arkangel episode of dystopian science fiction television series Black Mirror (Endemol Shine UK Ltd., 2017) in which parents equip their children with anti-violence visual filters via a brain implant.…”