Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

Abstract

Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. However, such observations have never been made formal or quantitative. In fact, it is not even clear whether the difficulties arising from a poorly chosen noise distribution are statistical or algorithmic in nature. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used. Namely, we prove these challenges arise due to an ill-behaved (more precisely, flat) loss landscape. To address this, we introduce a variant of NCE called eNCE which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.

Statistical applications of contrastive learning

“…The exponential loss is, like the logistic loss, a Bregman divergence that was shown to provide a large family of loss functions for the contrastive learning of energy-based models (Gutmann and Hirayama 2011). The result by Liu et al (2021) highlights that for some models, particular instances of the family of loss functions are more suitable than others.…”

“…Telescoping density-ratio estimation by Rhodes et al (2020), Choi et al ( 2021) deals with the density chasm and the flat loss landscape by re-formulating the contrastive problem. A complementary algorithmic approach was taken by Liu et al (2021) who asked whether optimisation techniques can deal with the flat optimisation landscape of the logistic loss when there is a density chasm. They found that, in case of exponential families, normalised gradient descent can deal with the issue.…”