Exploiting Eigenposteriors for Semi-Supervised Training of DNN Acoustic Models with Sequence Discrimination

Abstract

Deep neural network (DNN) acoustic models yield posterior probabilities of senone classes. Recent studies support the existence of low-dimensional subspaces underlying senone posteriors. Principal component analysis (PCA) is applied to identify eigenposteriors and perform low-dimensional projection of the training data posteriors. The resulted enhanced posteriors are applied as soft targets for training better DNN acoustic model under the student-teacher framework. The present work advances this approach by studying incorporation of sequence discriminative training. We demonstrate how to combine the gains from eigenposterior based enhancement with sequence discrimination to improve ASR using semi-supervised training. Evaluation on AMI meeting corpus yields nearly 4% absolute reduction in word error rate (WER) compared to the baseline DNN trained with cross entropy objective. In this context, eigenposterior enhancement of the soft targets is crucial to enable additive improvement using out-of-domain untranscribed data.

Far-Field ASR Using Low-Rank and Sparse Soft Targets from Parallel Data

“…The success of sparse soft targets for SDM shows that the non-linear low-dimensional modeling of senone subspaces, enabled by dictionaries, is highly beneficial for mapping reverberated noisy speech acoustic features to underlying senone classes. We also note that the performance improvements using enhanced soft targets are observed in both CE and sMBR loss based systems, and we conclude that the benefits of enhanced soft targets are complementary to those of sequence training, as shown previously in [9].…”

“…In terms of CS theory [11,23], while we take measurements in a very high dimensional DNN posterior space, the actual subspace where each senone belongs is very low-dimensional. Our earlier works [8,9,10, ?] on acoustic modeling explicitly took into account this multi-subspace structure of the speech data.…”

“…On the other hand, the information content is manifested in the form of structured patterns visible among a large population of training data posterior probabilities. In [9] and [10], we showed that low-rank reconstruction using eigenposteriors (class specific principal components in DNN posterior space) and sparse reconstruction using compressive sensing (CS) dictionaries [11] are principled ways of preserving the global low-dimensional structures in soft targets while discarding the random highdimensional noise. Soft targets enhanced in this way were successfully employed to improve close-talk ASR perfor-mance in our previous works [8,9].…”

“…In [9] and [10], we showed that low-rank reconstruction using eigenposteriors (class specific principal components in DNN posterior space) and sparse reconstruction using compressive sensing (CS) dictionaries [11] are principled ways of preserving the global low-dimensional structures in soft targets while discarding the random highdimensional noise. Soft targets enhanced in this way were successfully employed to improve close-talk ASR perfor-mance in our previous works [8,9]. We will revisit more details in Section 2, but in summary, low rank and sparse soft targets are high quality substitutes to replace hard senone alignments for training DNN acoustic models as they provide an improved mapping of the acoustic features to the underlying senone classes for hybrid DNN-HMM ASR.…”