Nebula: F0 Estimation and Voicing Detection by Modeling the Statistical Properties of Feature Extractors

Abstract

A F0 and voicing status estimation algorithm for high quality speech analysis/synthesis is proposed. This problem is approached from a different perspective that models the behavior of feature extractors under noise, instead of directly modeling speech signals. Under time-frequency locality assumptions, the joint distribution of extracted features and target F0 can be characterized by training a bank of Gaussian mixture models (GMM) on artificial data generated from Monte-Carlo simulations. The trained GMMs can then be used to generate a set of conditional distributions on the predicted F0, which are then combined and post-processed by Viterbi algorithm to give a final F0 trajectory. Evaluation on CSTR and CMU Arctic speech databases shows that the proposed method, trained on fully synthetic data, achieves lower gross error rates than state-of-the-art methods.

Data Augmentation Strategies for Neural Network F0 Estimation

“…The 1D convolution operation within the module is performed with varying levels of dilations over the frame time axis to model the time structure of F0 contours. In this study, the used dilations for the 8-layer residual module stack are d = [1,2,4,8,1,2,4,8], yielding (including the postnet) a receptive field of 71 frames with the selected filter length of 5 (current frame plus 35 past frames and 35 future frames) that condition each F0 estimate. The output of the convolution is passed through tanh activation and multiplied by a gating activation produced by a similar operation with the logistic sigmoid activation function.…”

“…Frequency domain methods utilize, for example, the energy of the linear prediction residual harmonics (e.g., SRH [4]) or instantaneous frequency (e.g., TEMPO [5]). In addition to the methods that provide raw frame-level estimates of F0, multiple methods have been developed for candidate F0 selection and/or postprocessing of the raw F0 estimates for improved robustness (e.g., pYIN [6], YAAPT [7], and Nebula [8]).…”