Selecting β-Divergence for Nonnegative Matrix Factorization by Score Matching

Lu, Zhiyun; Yang, Zhirong; Oja, Erkki

doi:10.1007/978-3-642-33266-1_52

Cited by 8 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we demonstrate the proposed method on various data types and learning tasks. First we provide the results on synthetic data, whose density is known, to compare the behavior of MTL, MEDAL and the score matching method [29]. Second, we illustrate the advantage of the EDA density over ED.…”

Section: Methodsmentioning

confidence: 99%

“…(21). The loglikelihood and negative score matching objectives [29] on the same four datasets are shown. The estimates are consistent with the ground truth Gaussian and Poisson data.…”

Section: Methodsmentioning

confidence: 99%

“…Finally, let us note that in addition to the maximum likelihood estimator, Score Matching (SM) [25], [37] can be applied to estimation of β as a density parameter (see Section IV-A). In a previous effort, Lu et al [29] proposed a similar exponential divergence (ED) distribution…”

Section: ) Maximum Exponential Divergence With Augmentation Likelihomentioning

confidence: 99%

See 2 more Smart Citations

Learning the Information Divergence

Dikmen

Yang

Oja

2015

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems. Examples are Nonnegative Matrix/Tensor Factorization, Stochastic Neighbor Embedding, topic models, and Bayesian network optimization. The success of such a learning task depends heavily on a suitable divergence. A large variety of divergences have been suggested and analyzed, but very few results are available for an objective choice of the optimal divergence for a given task. Here we present a framework that facilitates automatic selection of the best divergence among a given family, based on standard maximum likelihood estimation. We first propose an approximated Tweedie distribution for the β-divergence family. Selecting the best β then becomes a machine learning problem solved by maximum likelihood. Next, we reformulate α-divergence in terms of β-divergence, which enables automatic selection of α by maximum likelihood with reuse of the learning principle for β-divergence. Furthermore, we show the connections between γ- and β-divergences as well as Renyi- and α-divergences, such that our automatic selection framework is extended to non-separable divergences. Experiments on both synthetic and real-world data demonstrate that our method can quite accurately select information divergence across different learning problems and various divergence families.

show abstract

Section: Methodsmentioning

confidence: 99%

“…(21). The loglikelihood and negative score matching objectives [29] on the same four datasets are shown. The estimates are consistent with the ground truth Gaussian and Poisson data.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Learning the Information Divergence

Dikmen

Yang

Oja

2015

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…One of such metrics for the STFT spectra is Itakura-Saito divergence [21], and it was applied to nonnegative matrix factorization (NMF) [22,23] with successful results in music sound analysis [22]. β-divergence also provides scale-invariant distance, and it was applied to NMF as well [24,25]. However, in DNN learning, it is preferred to use simple metrics that are easy to differentiate to derive a learning algorithm and compute gradients efficiently.…”

Section: Loss Function Of Singing Voice Separation Modelsmentioning

confidence: 99%

Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks

2020

View full text Add to dashboard Cite

Single-channel singing voice separation has been considered a difficult task, as it requires predicting two different audio sources independently from mixed vocal and instrument sounds recorded by a single microphone. We propose a new singing voice separation approach based on the curriculum learning framework, in which learning is started with only easy examples and then task difficulty is gradually increased. In this study, we regard the data providing obviously dominant characteristics of a single source as an easy case and the other data as a difficult case. To quantify the dominance property between two sources, we define a dominance factor that determines a difficulty level according to relative intensity between vocal sound and instrument sound. If a given data is determined to provide obviously dominant characteristics of a single source according to the factor, it is regarded as an easy case; otherwise, it belongs to a difficult case. Early stages in the learning focus on easy cases, thus allowing rapidly learning overall characteristics of each source. On the other hand, later stages handle difficult cases, allowing more careful and sophisticated learning. In experiments conducted on three song datasets, the proposed approach demonstrated superior performance compared to the conventional approaches.

show abstract

“…The resulting algorithms have been successfully applied to various learning tasks such as part-based representation learning, clustering, bi-clustering, and graph matching [18][19][20][21][22][23][24][25][26][27][28][29][30].…”

mentioning

confidence: 99%