Contribution of modulation spectral features on the perception of vocal-emotion using noise-vocoded speech

et al. 2020

IEEE Access

Self Cite

Emotion information from speech can effectively help robots understand speaker's intentions in natural human-robot interaction. The human auditory system can easily track temporal dynamics of emotion by perceiving the intensity and fundamental frequency of speech, and focus on the salient emotion regions. Therefore, speech emotion recognition combined with the auditory mechanism and attention mechanism may be an effective way. Some previous studies used auditory-based static features to identify emotion while ignoring the emotion dynamics. Some other studies used attention models to capture the salient regions of emotion while ignoring cognitive continuity. To fully utilize the auditory and attention mechanism, we first investigate temporal modulation cues from auditory front-ends and then propose a joint deep learning model that combines 3D convolutions and attention-based sliding recurrent neural networks (ASRNNs) for emotion recognition. Our experiments on the IEMOCAP and MSP-IMPROV datasets indicate that the proposed method can be effectively used to recognize the emotions of speech from temporal modulation cues. The subjective evaluation shows that the attention patterns of the attention model are basically consistent with human behaviors in recognizing the emotions.

Section: Introductionmentioning

confidence: 99%

Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks With Auditory Front-Ends

Peng

et al. 2020

IEEE Access

Self Cite

“…We previously reported that temporal modulationspectral features derived from the temporal-modulation spectrogram (time, acoustical frequency, and modulation frequency domains), as shown in Figs. 6 and 7, can be clarified to categorize emotion types according to the discriminability index (d 0 ) in vocal-emotion recognition [42]. In particular, higher-order moments, such as modulation spectral centroid (MSCR), modulation spectral spread (MSSP), modulation spectral skewness (MSSK), modulation spectral kurtosis (MSKT), and modulation spectral tilt (MSTL), are useful modulation-spectral features.…”

Section: Temporal Modulation-spectral Featuresmentioning

confidence: 99%

“…Estimated probability-distribution function of MSCR (MSCR m , m ¼ 4) under clean conditions[42]. Similarity of each modulation-spectral feature (taken across all acoustic or modulation frequency channels)[42].…”

mentioning

confidence: 99%

Relationship between contributions of temporal amplitude envelope of speech and modulation transfer function in room acoustics to perception of noise-vocoded speech

Unoki

2020

Acoust. Sci. & Tech.

Self Cite

Speech signals can be represented as a sum of amplitude-modulated frequency bands. This sum can also be regarded as a temporal amplitude envelope (TAE) with temporal fine structure. Our previous studies using noise-vocoded speech (NVS) showed that the TAE of speech plays an important role in the perception of linguistic information (speech intelligibility) as well as non-linguistic information (e.g., vocal-emotion recognition). It was found that the upper limit of the modulation frequency from 4 to 8 Hz on the TAE is important for speech intelligibility, while that from 8 to 16 Hz is important for vocal-emotion recognition. However, speech intelligibility generally dramatically degrades due to reverberation. The concept of the modulation transfer function (MTF) takes into account the relationship between the transfer function in an enclosure in terms of input and output TAEs and characteristics of the enclosure under reverberant conditions. This concept was introduced as a measure in room acoustics for assessing the effect of an enclosure on speech intelligibility. For this study, we conducted two experiments involving word intelligibility tests and vocal-emotion recognition with NVS under reverberant conditions to investigate the relationship between the contributions of the TAE of speech and MTF of reverberation to modulation perception of NVS. We also pointed out that the straightforward scheme, i.e., the relationship between the contributions of the static features (peak/slope) in the modulation spectrum (MS) of speech and MTF of reverberation, cannot consistently account for the auditory perception of both linguistic and non-linguistic information obtained from these perceptual data of NVS under reverberant conditions. We then developed a scheme in which the relationship between the contributions of the temporal MS features and MTF of reverberation to modulation perception can consistently account for these perceptual data of NVS.

“…Studies revealed that TAEs of speech and their modulation spectral components contribute to the recognition of linguistic information as well as perception of non-linguistic information. We previously investigated the effect of controlling the upper limit of the modulation frequency of a TAE on the perception of non-linguistic information, i.e., speaker individuality and vocalemotion, using NVS [7][8][9]. It was found that the speakerdistinction and vocal-emotion-recognition rates decrease as the upper limit of the modulation frequency becomes lower [7][8][9].…”

Section: Introductionmentioning

confidence: 99%

“…We previously investigated the effect of controlling the upper limit of the modulation frequency of a TAE on the perception of non-linguistic information, i.e., speaker individuality and vocalemotion, using NVS [7][8][9]. It was found that the speakerdistinction and vocal-emotion-recognition rates decrease as the upper limit of the modulation frequency becomes lower [7][8][9].…”

Section: Introductionmentioning

confidence: 99%

Study on the perception of nonlinguistic information of noise-vocoded speech under noise and/or reverberation conditions

Kawamura

Unoki

2022

Acoust. Sci. & Tech.

Self Cite

It has been known that noise and reverberation greatly affect the perception of linguistic information, in particular speech intelligibility. However, the effect of noise and reverberation on the perception of non-linguistic information has not been clarified. We investigated how these types of disturbances affect the perception of non-linguistic information (speaker individuality and vocal emotion) of noise-vocoded speech. We conducted speaker-distinction and vocal-emotion-recognition experiments using noise-vocoded speech created from the speech in noisy, reverberation, and noisy reverberant environments as stimuli. We used seven noise conditions (signal-to-noise ratio (SNR) = 1; 20; 15; 10; 5; 0; À5 dB) and six reverberation conditions (reverberation time (T R ) = 0:0; 0:1; 0:3; 0:5; 1:0; 2:0 s). In both speaker-distinction and vocal-emotion-recognition experiments, the main effects of noise and reverberation were significant, but the interaction was not significant. From these results, except for extremely poor sound conditions, under daily noise and reverberation conditions (an SNR of more than 10 dB and T R less than 1.0 s), there were no significant effects of noise and reverberation.