Density Ratio Estimation via Infinitesimal Classification

Abstract

Density ratio estimation (DRE) is a fundamental machine learning technique for comparing two probability distributions. However, existing methods struggle in high-dimensional settings, as it is difficult to accurately compare probability distributions based on finite samples. In this work we propose DRE-8, a divide-and-conquer approach to reduce DRE to a series of easier subproblems. Inspired by Monte Carlo methods, we smoothly interpolate between the two distributions via an infinite continuum of intermediate bridge distributions. We then estimate the instantaneous rate of change of the bridge distributions indexed by time (the "time score")-a quantity defined analogously to data (Stein) scores-with a novel time score matching objective. Crucially, the learned time scores can then be integrated to compute the desired density ratio. In addition, we show that traditional (Stein) scores can be used to obtain integration paths that connect regions of high density in both distributions, improving performance in practice. Empirically, we demonstrate that our approach performs well on downstream tasks such as mutual information estimation and energy-based modeling on complex, high-dimensional datasets.

Statistical applications of contrastive learning

“…The functional form or a model of the auxiliary distributions does not need to be known since the method only requires samples from them. It is further possible to derive a limiting objective function when the number of auxiliary distributions K goes to infinity, which was shown to lead to improved performance and removes the need to choose K (Choi et al 2021).…”

“…property were proposed by Pihlaja et al (2010), Hirayama (2011), andUehara et al (2020). Moreover, the telescoping approach by Rhodes et al (2020), andChoi et al (2021) can also be used to learn h, improving upon the single-ratio methods. For an overview of further methods to estimate energy-based models, we refer the reader to the recent review by Song and Kingma (2021), and for the case of energy-based models with latent (unobserved) variables to the work by Rhodes and Gutmann (2019).…”