Noise contrastive estimation: Asymptotic properties, formal comparison with MC-MLE

Abstract

A statistical model is said to be un-normalised when its likelihood function involves an intractable normalising constant. Two popular methods for parameter inference for these models are MC-MLE (Monte Carlo maximum likelihood estimation), and NCE (noise contrastive estimation); both methods rely on simulating artificial data-points to approximate the normalising constant. While the asymptotics of MC-MLE have been established under general hypotheses (Geyer, 1994), this is not so for NCE. We establish consistency and asymptotic normality of NCE estimators under mild assumptions. We compare NCE and MC-MLE under several asymptotic regimes. In particular, we show that, when m → ∞ while n is fixed (m and n being respectively the number of artificial data-points, and actual data-points), the two estimators are asymptotically equivalent. Conversely, we prove that, when the artificial data-points are IID, and when n → ∞ while m/n converges to a positive constant, the asymptotic variance of a NCE estimator is always smaller than the asymptotic variance of the corresponding MC-MLE estimator. We illustrate the variance reduction brought by NCE through a numerical study.

Statistical applications of contrastive learning

“…where where have used that for ν → ∞, m = νn becomes arbitrarily large too, so that the average over the y i becomes an expectation with respect to q(y). A more general result was established by Riou-Durand and Chopin (2018). Moreover, they further considered the case of finite ν, but large sample sizes n, and showed that the variance of the noise-contrastive estimator is always smaller than the variance of the Monte Carlo MLE estimator (Geyer, 1994) where the partition function is approximated with a sample average, assuming in both cases that the auxiliary/reference distributions were fixed.…”

Statistical applications of contrastive learning

“…, and taking the limit of → ∞ in (38), thus, gives where where have used that for → ∞ , m = n becomes arbitrarily large too, so that the average over the i becomes an expectation with respect to q( ) . A more general result was established by Riou-Durand and Chopin (2018). Moreover, they further considered the case of finite , but large sample sizes n, and showed that the variance of the noise-contrastive estimator is always smaller than the variance of the Monte Carlo MLE estimator (Geyer 1994) where the partition function is approximated with a sample average, assuming in both cases that the auxiliary/reference distributions were fixed.…”

Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

“…With respect to a fixed Q, it remains an open question about what formally are the nature of the challenges posed by a poorly chosen Q, which could be statistical and/or algorithmic. Various previous works have analyzed the asymptotic behavior of NCE and its variants (Gutmann & Hyvärinen, 2012;Riou-Durand et al, 2018;Uehara et al, 2020), but these do not provide guidance on the finite sample behavior of NCE or its common variants. The improvements to NCE in prior works are all borne out by the empirical observations of NCE practitioners, rather than motivated by theory, which is precisely the aim of this work.…”