Application of neural networks in whispered speech recognition

Grozdic, Dorde T.; Markovic, Branko; Galic, Jovan; Jovicie, Slobodan T.

doi:10.1109/telfor.2012.6419311

Cited by 9 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This degradation is not as pronounced if an AM trained from whispered speech were to be used to recognize neutral speech. This result has been consistent for experiments done in several languages for which there is a sizeable corpus of parallel whispered speech, such as Serbian [3], Japanese [4], Mandarin [5] and English [6,7]. The pattern is also consistent for different types of models, both Gaussian Mixture Models (GMM) trained with generative or discriminative methods, and even for Deep Neural Network (DNN) based AMs.…”

Section: Related Worksupporting

confidence: 81%

Transfer Learning with Bottleneck Feature Networks for Whispered Speech Recognition

Lim

Wong²,

Li³

et al. 2016

Interspeech 2016

View full text Add to dashboard Cite

Previous work on whispered speech recognition has shown that acoustic models (AM) trained on whispered speech can somewhat classify unwhispered (neutral) speech sounds, but not vice versa. In fact, AMs trained purely on neutral speech completely fail to recognize whispered speech. Meanwhile, recipes used to train neutral AMs will work just as well for whispered speech, but such methods require a large volume of transcribed whispered speech which is expensive to gather. In this work, we propose and investigate the use of bottleneck feature networks to normalize differences between whispered and neutral speech modes. Our extensive experiments show that this type of speech variability can be effectively normalized. We also show that it is possible to transfer this knowledge from two source languages with whispered speech (Mandarin and English), to a new target language (Malay) without whispered speech. Furthermore, we report a substantial reduction in word error rate for cross-mode speech recognition, effectively demonstrate that it is possible to train acoustic models capable of classifying both types of speech without needing any additional whispered speech.

show abstract

Section: Related Worksupporting

confidence: 81%

Transfer Learning with Bottleneck Feature Networks for Whispered Speech Recognition

Lim

Wong²,

Li³

et al. 2016

Interspeech 2016

View full text Add to dashboard Cite

show abstract

“…However, the effectiveness of E2E approaches over whispered speech is yet to be confirmed. Previous works suggested that deep learning was useful for whispered speech recognition [21,22,23], while the success of E2E approaches on normal speech ASR was widely believed to depend on the quantity of data [24] and the model architecture [25]. It is much more difficult to collect whispered speech data of reasonable size, and the special characteristics of whispered speech may need special considerations in model design and training.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Whispered Speech Recognition with Frequency-Weighted Approaches and Pseudo Whisper Pre-training

Chang

Liu

Lee

et al. 2021

2021 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Whispering is an important mode of human speech, but no endto-end recognition results for it were reported yet, probably due to the scarcity of available whispered speech data. In this paper, we present several approaches for end-to-end (E2E) recognition of whispered speech considering the special characteristics of whispered speech and the scarcity of data. This includes a frequency-weighted SpecAugment policy and a frequencydivided CNN feature extractor for better capturing the high frequency structures of whispered speech, and a layer-wise transfer learning approach to pre-train a model with normal speech then fine-tuning it with whispered speech to bridge the gap between whispered and normal speech. We achieve an overall relative reduction of 19.8% in PER and 31.9% in CER on a relatively small whispered TIMIT corpus. The results indicate as long as we have a good E2E model pre-trained on normal speech, a relatively small set of whispered speech may suffice to obtain a reasonably good E2E whispered speech recognizer.

show abstract

“…Whispering is a natural mode of speech communication which in recent years has garnered more attention in the speech community [1,2,3,4,5,6]. During whisper, vocal folds do not vibrate and consequently pitch is absent [7].…”

Section: Introductionmentioning

confidence: 99%

Perception of Tone in Whispered Mandarin Sentences: The Case for Singapore Mandarin

Lim

Chen

2016

Interspeech 2016

View full text Add to dashboard Cite

Whispering is commonly used when one needs to speak softly (for instance, in a library). Whispered speech mainly differs from neutral speech in that voicing, and thus its acoustic correlate F0, is absent. It is well known that in tonal languages such as Mandarin, tone identity is primarily conveyed by the F0 contour. Previous works also suggest that secondary correlates are both consistent and sufficient to convey Mandarin tone in whisper. However, these results are focused on Standard Mandarin spoken in Mainland China and have only been obtained via small-scale experiments using citation-form speech. To investigate whether these results will carry over to continuous sentences in other variations of Mandarin, we present a study that is the first of its nature to explore native Singapore Mandarin. Unlike related works, our large-scale perceptual experiment thoroughly investigates lexical tones in whispered and neutral Mandarin by involving more diverse speech data, greater number of listeners and use syllables excised from continuous speech to better simulate natural speech conditions. Our findings differ significantly from earlier works in terms of the recognition patterns observed. We present further in-depth analysis on how various phonetic characteristics (vowel contexts, place and manner of articulation) affect whispered tone perception.

show abstract

Application of neural networks in whispered speech recognition

Cited by 9 publications

References 6 publications

Transfer Learning with Bottleneck Feature Networks for Whispered Speech Recognition

Transfer Learning with Bottleneck Feature Networks for Whispered Speech Recognition

End-to-End Whispered Speech Recognition with Frequency-Weighted Approaches and Pseudo Whisper Pre-training

Perception of Tone in Whispered Mandarin Sentences: The Case for Singapore Mandarin

Contact Info

Product

Resources

About