On The Sample Complexity of HGR Maximal Correlation Functions

Abstract

The Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and the corresponding functions have been shown useful in many machine learning scenarios. In this paper, we study the sample complexity of estimating the HGR maximal correlation functions by the alternative conditional expectation (ACE) algorithm from a sequence of training data in the asymptotic regime. Specifically, we develop a mathematical framework to characterize the learning errors between the maximal correlation functions computed from the true distribution, and the functions estimated from the ACE algorithm. For both supervised and semi-supervised learning scenarios, we establish the analytical expressions for the error exponents of the learning errors, which indicate the number of training samples required for estimating the HGR maximal correlation functions by the ACE algorithm. Moreover, with our theoretical results, we investigate the sampling strategy for different types of samples in semi-supervised learning with a total sampling budget constraint, and an optimal sampling strategy is developed to maximize the error exponent of the learning error. Finally, the numerical simulations are presented to support our theoretical results.

On Universal Features for High-Dimensional Learning and Inference

“…Corollary 84: Let X, Y be zero-mean jointly Gaussian variables characterized by Λ X , Λ Y , and Λ XY , and let U and V be variables in the Gauss-Markov chain (301) such that we satisfy the independence relations Λ U = Λ V = I, the conditional independence relations that 57…”

“…Proposition 86: Let X ∈ R K X , Y ∈ R K Y be jointly Gaussian variables characterized by Λ X , Λ Y , and Λ XY , and 57 The elements of U are conditionally independent given X when…”