Acoustic to articulatory mapping with deep neural network

Wu, Zhiyong; Zhao, Kai; Wu, Xixin; Lan, Xinyu; Meng, Helen

doi:10.1007/s11042-014-2183-z

Cited by 21 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The relation between the acoustic features and articulatory movements is known to be non-linear and non-unique [14,3].…”

Section: Proposed Approachmentioning

confidence: 99%

“…There are several applications of AAI including speech recognition [2,3], speech synthesis [4], speaker verification [5] and multimedia applications [6,7,8]. For subject dependent AAI (SD-AAI), various approaches have been proposed in the literature including codebook [9,10], Gaussian mixture model (GMM) [11], Hidden Markov Model (HMM) [12], mixture trajectory model [13], Deep Neural Network (DNN) [14,15,16]. All these approaches need parallel acoustic-articulatory data for training AAI model, which, in turn, requires recording of speech and simultaneous motion of articulators from a subject of interest.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Low Resource Acoustic-to-articulatory Inversion Using Bi-directional Long Short Term Memory

Illa

Ghosh

2018

Interspeech 2018

View full text Add to dashboard Cite

Estimating articulatory movements from speech acoustic features is known as acoustic-to-articulatory inversion (AAI). Large amount of parallel data from speech and articulatory motion is required for training an AAI model in a subject dependent manner, referred to as subject dependent AAI (SD-AAI). Electromagnetic articulograph (EMA) is a promising technology to record such parallel data, but it is expensive, time consuming and tiring for a subject. In order to reduce the demand for parallel acoustic-articulatory data in the AAI task for a subject, we, in this work, propose a subject-adaptative AAI method (SA-AAI) from an existing AAI model which is trained using large amount of parallel data from a fixed set of subjects. Experiments are performed with 30 subjects' acoustic-articulatory data and AAI is trained using BLSTM network to examine the amount of data needed from a new target subject for the SA-AAI to achieve an AAI performance equivalent to that of SD-AAI. Experimental results reveal that the proposed SA-AAI performs similar to that of the SD-AAI with ∼62.5% less training data. Among different articulators, the SA-AAI performance for tongue articulators matches with the corresponding SD-AAI performance with only ∼12.5% of the data used for SD-AAI training.

show abstract

“…The relation between the acoustic features and articulatory movements is known to be non-linear and non-unique [14,3].…”

Section: Proposed Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Low Resource Acoustic-to-articulatory Inversion Using Bi-directional Long Short Term Memory

Illa

Ghosh

2018

Interspeech 2018

View full text Add to dashboard Cite

show abstract

“…On the other hand, the deep neural network can extract more representative features from the raw data in a pre-training way for obtaining more accurate prediction results. Due to the superiority in feature extraction and model fitting, deep learning has attracted a great amount of attention around the world, and has been widely applied in various fields, such as green buildings [27,28], image processing [29][30][31][32], speech recognition [33,34], and intelligent traffic management systems [35][36][37]. As a novel deep learning method, the long-short-term memory network (LSTM) can make full use of the historical information due to its special structure [38].…”

Section: Introductionmentioning

confidence: 99%

Long-Short-Term Memory Network Based Hybrid Model for Short-Term Electrical Load Forecasting

Xie

et al. 2018

Information

View full text Add to dashboard Cite

Short-term electrical load forecasting is of great significance to the safe operation, efficient management, and reasonable scheduling of the power grid. However, the electrical load can be affected by different kinds of external disturbances, thus, there exist high levels of uncertainties in the electrical load time series data. As a result, it is a challenging task to obtain accurate forecasting of the short-term electrical load. In order to further improve the forecasting accuracy, this study combines the data-driven long-short-term memory network (LSTM) and extreme learning machine (ELM) to present a hybrid model-based forecasting method for the prediction of short-term electrical loads. In this hybrid model, the LSTM is adopted to extract the deep features of the electrical load while the ELM is used to model the shallow patterns. In order to generate the final forecasting result, the predicted results of the LSTM and ELM are ensembled by the linear regression method. Finally, the proposed method is applied to two real-world electrical load forecasting problems, and detailed experiments are conducted. In order to verify the superiority and advantages of the proposed hybrid model, it is compared with the LSTM model, the ELM model, and the support vector regression (SVR). Experimental and comparison results demonstrate that the proposed hybrid model can give satisfactory performance and can achieve much better performance than the comparative methods in this short-term electrical load forecasting application.

show abstract

“…After several rounds of review, ten papers were finally selected to be included in this special issue. These papers can be categorized into three topics: avatar animation [3,9,13], speech synthesis [11,12,16,18] and human emotion/behavior analysis [5,10,17].…”

mentioning

confidence: 99%

“…Significant performance gain in head motion prediction is reported. Taking the advantage of the rich non-linear learning ability, Wu et al [13] develop a DNN approach for real-time speech driven talking avatar. Specifically, the input of the system is acoustic speech and the output is articulatory movements on a three-dimensional avatar.…”

mentioning

confidence: 99%

Expressive talking avatar synthesis and animation

Xie

Jia

Meng

et al. 2015

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

The talking avatar, an animated speaking virtual character with vivid human-like appearance and real or synthetic speech, has gradually shown its potential in applications involving human-computer intelligent interactions. The talking avatar has rich communication abilities to deliver verbal and nonverbal information by voice, tones, eye-contact, head motion and facial expressions, etc. Avatars are increasingly being used on a variety of electronic devices, such as computers, smart phones, pads, kiosks and game consoles. Avatars also can be found across many domains, such as technical support and customer service, communication aids, speech therapy, virtual reality, film special effects, education and training [6]. Specific applications may include a virtual storyteller for children, a virtual guider or presenter for personal or commercial website, a representative of user in computer games and a funny puppetry for computer-mediated human communications. It is clearly promising that talking avatars will become an expressive multimodal interface in human computer interaction.

show abstract

Acoustic to articulatory mapping with deep neural network

Cited by 21 publications

References 29 publications

Low Resource Acoustic-to-articulatory Inversion Using Bi-directional Long Short Term Memory

Low Resource Acoustic-to-articulatory Inversion Using Bi-directional Long Short Term Memory

Long-Short-Term Memory Network Based Hybrid Model for Short-Term Electrical Load Forecasting

Expressive talking avatar synthesis and animation

Contact Info

Product

Resources

About