Probabilistic Latent Variable Models as Nonnegative Factorizations

Abstract

This paper presents a family of probabilistic latent variable models that can be used for analysis of nonnegative data. We show that there are strong ties between nonnegative matrix factorization and this family, and provide some straightforward extensions which can help in dealing with shift invariances, higher-order decompositions and sparsity constraints. We argue through these extensions that the use of this approach allows for rapid development of complex statistical models for analyzing nonnegative data.

Automatic Transcription of Polyphonic Vocal Music

“…Those state hypotheses with an entirely correct voice assignment are represented by a thick border. [41,3] in O 3 . The two most likely state hypotheses at each step are listed in the large rectangles above each state S t , where the voices are notated with braces.…”

“…(Notes belonging to the upper voice are shown in red and notes belonging to the lower voice are shown in blue.) Again, notice false positive pitch detection [41,3]. In this figure, the emission sets O t are shown on the bottom, and the boxes below each O t node list the emitted notes in decreasing pitch order.…”

“…These approaches decompose an input time-frequency representation (such as a spectrogram) into a linear combination of non-negative factors, often consisting of spectral atoms and note activations. The most successful of these spectrogram factorization methods have been based on non-negative matrix factorisation (NMF) [2] or probabilistic latent component analysis (PLCA) [3].…”

Low Rank Language Models for Small Training Sets

“…The connection between non-negative tensor factorization and latent variable models has been previously explored in the literature (e.g., in [7], [8]). Non-negative tensor factorization models have also been applied to other language processing applications, including subject-verb-object selectional preference induction [9] and learning semantic word similarity [10].…”

“…Formally, we find a local solution to the problem: (8) where denotes the set of element-wise non-negative tensors whose entries sum to one, i.e., the set of tensors corresponding to valid joint probability distributions; are the -grams in the training data (obtained by sliding a window of size over each sentence); and is the probability distribution given by . 1 For traditional -gram models, the maximum likelihood objective yields models that are highly overfit to the data; in particular, they are plagued with zero probability -grams.…”