Evaluating speech features with the minimal-pair ABX task (II): resistance to noise

Schatz, Thomas; Peddinti, Vijayaditya; Cao, Xuan-Nga; Bach, Francis; Heřmanský, Hynek; Dupoux, Emmanuel

doi:10.21437/interspeech.2014-228

Cited by 23 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By highlighting how the gradient computation separates into two steps we derived an importancesampling estimate for the gradient that often only needs to evaluate the computationally cheaper part to provide the estimate. Skipping the computationally costly evaluation of the gradient of the model itself as often as possible lead to a practical speedup that is independent of other improvements provided by more advanced optimization algorithms (Johnson and Zhang, 2013;Schatz et al, 2014). Our method relies on the inverse transformation being unique and efficient to compute.…”

Section: Discussionmentioning

confidence: 99%

“…In recent years several more advanced stochastic optimization algorithms have been proposed, such as stochastic average gradients (SAG) (Schmidt et al, 2017), stochastic variance reduced gradients (SVRG) (Johnson and Zhang, 2013), and SAGA that combines elements of both (Schatz et al, 2014). However, to our knowledge these techniques have not been successfully adapted for automatic variational inference.…”

Section: Stochastic Gradient Optimizationmentioning

confidence: 99%

“…The full gradient estimate hence consists of individual gradients estimated for different parameter values; the most recently computed gradients are accurate but the ones computed long time ago may correspond to vastly different parameter values. This introduces bias (Schatz et al, 2014), but especially towards the convergence the variance of the estimated full batch gradient is considerably smaller than that of the latest mini-batch, speeding up convergence.…”

Section: Stochastic Gradient Optimizationmentioning

confidence: 99%

See 2 more Smart Citations

Importance Sampled Stochastic Optimization for Variational Inference

Sakaya,

Klami

2017

Preprint

View full text Add to dashboard Cite

Variational inference approximates the posterior distribution of a probabilistic model with a parameterized density by maximizing a lower bound for the model evidence. Modern solutions fit a flexible approximation with stochastic gradient descent, using Monte Carlo approximation for the gradients. This enables variational inference for arbitrary differentiable probabilistic models, and consequently makes variational inference feasible for probabilistic programming languages. In this work we develop more efficient inference algorithms for the task by considering importance sampling estimates for the gradients. We show how the gradient with respect to the approximation parameters can often be evaluated efficiently without needing to re-compute gradients of the model itself, and then proceed to derive practical algorithms that use importance sampled estimates to speed up computation. We present importance sampled stochastic gradient descent that outperforms standard stochastic gradient descent by a clear margin for a range of models, and provide a justifiable variant of stochastic average gradients for variational inference.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Stochastic Gradient Optimizationmentioning

confidence: 99%

Section: Stochastic Gradient Optimizationmentioning

confidence: 99%

See 1 more Smart Citation

Importance Sampled Stochastic Optimization for Variational Inference

Sakaya,

Klami

2017

Preprint

View full text Add to dashboard Cite

show abstract

“…To test if the learned representations can separate phonetic categories, we use a minimal pair ABX discrimination task [20,21]. It only requires to define a dissimilarity function d between speech tokens, no external training algorithm is needed.…”

Section: Evaluation With Abx Tasksmentioning

confidence: 99%

Sampling Strategies in Siamese Networks for Unsupervised Speech Representation Learning

Riad¹,

Dancette²,

Karadayi³

et al. 2018

Interspeech 2018

Self Cite

View full text Add to dashboard Cite

Recent studies have investigated siamese network architectures for learning invariant speech representations using samedifferent side information at the word level. Here we investigate systematically an often ignored component of siamese networks: the sampling procedure (how pairs of same vs. different tokens are selected). We show that sampling strategies taking into account Zipf's Law, the distribution of speakers and the proportions of same and different pairs of words significantly impact the performance of the network. In particular, we show that word frequency compression improves learning across a large range of variations in number of training pairs. This effect does not apply to the same extent to the fully unsupervised setting, where the pairs of same-different words are obtained by spoken term discovery. We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on state-of-the-art in unsupervised representation learning using siamese networks.

show abstract

“…The evaluation metrics defined by the Zero Resource Speech Challenge of Interspeech 2015 [20] were used in this work. For evaluating the discovered frame-level features, the minimum pair ABX task [43], [44] was used to measure the discriminability between two sound categories as in Track 1 of the Challenge. For evaluating the discovered tokens in Track 2 of the Challenge, a total of seven evaluation metrics were defined: Normalized Edit Distance (NED), Coverage, Matching F-score, Grouping F-score, Type F-score, Token Fscore and Boundary F-score [20].…”

Section: B Evaluation Metrics For Discovered Features and Tokensmentioning

confidence: 99%

Unsupervised Iterative Deep Learning of Speech Features and Acoustic Tokens with Applications to Spoken Term Detection

Chung

Tsai

Liu

et al. 2017

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-In this paper we aim to automatically discover high quality frame-level speech features and acoustic tokens directly from unlabeled speech data. A Multi-granular Acoustic Tokenizer (MAT) was proposed for automatic discovery of multiple sets of acoustic tokens from the given corpus. Each acoustic token set is specified by a set of hyperparameters describing the model configuration. These different sets of acoustic tokens carry different characteristics for the given corpus and the language behind, thus can be mutually reinforced. The multiple sets of token labels are then used as the targets of a Multi-target Deep Neural Network (MDNN) trained on frame-level acoustic features. Bottleneck features extracted from the MDNN are then used as the feedback input to the MAT and the MDNN itself in the next iteration. The multi-granular acoustic token sets and the frame-level speech features can be iteratively optimized in the iterative deep learning framework. We call this framework the Multi-granular Acoustic Tokenizing Deep Neural Network (MAT-DNN). The results were evaluated using the metrics and corpora defined in the Zero Resource Speech Challenge organized at Interspeech 2015, and improved performance was obtained with a set of experiments of query-by-example spoken term detection on the same corpora. Visualization for the discovered tokens against the English phonemes was also shown.

show abstract

Evaluating speech features with the minimal-pair ABX task (II): resistance to noise

Cited by 23 publications

References 17 publications

Importance Sampled Stochastic Optimization for Variational Inference

Importance Sampled Stochastic Optimization for Variational Inference

Sampling Strategies in Siamese Networks for Unsupervised Speech Representation Learning

Unsupervised Iterative Deep Learning of Speech Features and Acoustic Tokens with Applications to Spoken Term Detection

Contact Info

Product

Resources

About