Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Abstract

Research on speech processing has traditionally considered the task of designing hand-engineered acoustic features (feature engineering) as a separate distinct problem from the task of designing efficient machine learning (ML) models to make prediction and classification decisions. There are two main drawbacks to this approach: firstly, the feature engineering being manual is cumbersome and requires human knowledge; and secondly, the designed features might not be best for the objective at hand. This has motivated the adoption of a recent trend in speech community towards utilisation of representation learning techniques, which can learn an intermediate representation of the input signal automatically that better suits the task at hand and hence lead to improved performance. The significance of representation learning has increased with advances in deep learning (DL), where the representations are more useful and less dependent on human knowledge, making it very conducive for tasks like classification, prediction, etc. The main contribution of this paper is to present an up-to-date and comprehensive survey on different techniques of speech representation learning by bringing together the scattered research across three distinct research areas including Automatic Speech Recognition (ASR), Speaker Recognition (SR), and Speaker Emotion Recognition (SER). Recent reviews in speech have been conducted for ASR, SR, and SER, however, none of these has focused on the representation learning from speech-a gap that our survey aims to bridge. We also highlight different challenges and the key characteristics of representation learning models and discuss important recent advancements and point out future trends. Our review can be used by the speech research community as an essential resource to quickly grasp the current progress in representation learning and can also act as a guide for navigating study in this area of research.

Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

“…With the advent of deep learning, there has been a shift from traditional human-crafted emotional features such as those extracted by low level descriptors (LLDs) [44] or openSMILE [45], to the features automatically learned by deep neural networks (DNN) [46]. Many studies [46,47] have shown that the deep features learned by DNN are more effective and thus more suitable for SER. Meanwhile, recent speech synthesis studies [48,49] also propose to leverage those deep emotional features to characterize different emotional styles over a continuum [50].…”

Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks

“…for downstream tasks. In speech representation learning (Latif et al, 2020), unsupervised techniques such as autoregressive modeling (Chung, Hsu, Tang and Glass, 2019;Chung and Glass, 2020a,b) and self-supervised modeling (Milde and Biemann, 2018;Tagliasacchi, Gfeller, Quitry and Roblek, 2019;Pascual, Ravanelli, Serrà, Bonafonte and Bengio, 2019) employ temporal context information for extracting speech representation. In our prior behavior modeling work, an unsupervised representative learning framework was proposed (Li, Baucom and Georgiou, 2017), which showed the promise of learning behavior representations based on the behavior stationarity hypothesis that nearby segments of speech share the same behavioral context.…”

“…Recently, unsupervised and self-supervised learning (Latif, Rana, Khalifa, Jurdak, Qadir and Schuller, 2020;Chen, Kornblith, Norouzi and Hinton, 2020) have shown the benefits of using large amounts of unlabelled data to extract informative representations. Given the low availability of annotated behavioral data sets, representation learning through unsupervised ways can provide a promising avenue for behavioral modeling.…”

Do Neural Network Weights account for Classes Centers?

“…In the first case, the output representation is usually obtained via a mathematical projection rule, like principal component analysis [5] and linear discriminant analysis [6], or a specific extraction scheme making use of a set of pre-defined handcrafted rules [7], [8], [9], [10]. In contrast, the second case employs a machine learning algorithm, in order to discover salient features from raw data [11], [12], [13].…”

“…Due to the complex nature of real-world sensory inputs in present problems, the manual definition of descriptive features becomes increasingly unreliable, leaving no option but to engage machine learning schemes. Typical algorithms are Support Vector Machines (SVMs) [11] and Deep Neural Networks (DNNs) [13], [14], with DNNs forming the leading choice in feature extraction for cascade [15], [16] and fusion tasks [17], [18] given their proven efficacy over the past years. Yet, in order for a learning algorithm to ensure robust representation capacity, a set of techniques needs to be applied during its training phase.…”