Universal and Composite Hypothesis Testing via Mismatched Divergence

Abstract

For the universal hypothesis testing problem, where the goal is to decide between the known null hypothesis distribution and some other unknown distribution, Hoeffding proposed a universal test in the nineteen sixties. Hoeffding's universal test statistic can be written in terms of Kullback-Leibler (K-L) divergence between the empirical distribution of the observations and the null hypothesis distribution. In this paper a modification of Hoeffding's test is considered based on a relaxation of the K-L divergence test statistic, referred to as the mismatched divergence. The resulting mismatched test is shown to be a generalized likelihood-ratio test (GLRT) for the case where the alternate distribution lies in a parametric family of the distributions characterized by a finite dimensional parameter, i.e., it is a solution to the corresponding composite hypothesis testing problem. For certain choices of the alternate distribution, it is shown that both the Hoeffding test and the mismatched test have the same asymptotic performance in terms of error exponents. A consequence of this result is that the GLRT is optimal in differentiating a particular distribution from others in an exponential family. It is also shown that the mismatched test has a significant advantage over the Hoeffding test in terms of finite sample size performance. This advantage is due to the difference in the asymptotic variances of the two test statistics under the null hypothesis. In particular, the variance of the K-L divergence grows linearly with the alphabet size, making the test impractical for applications involving large alphabet distributions. The variance of the mismatched divergence on the other hand grows linearly with the dimension of the parameter space, and can hence be controlled through a prudent choice of the function class defining the mismatched divergence.Comment: Accepted to IEEE Transactions on Information Theory, July 201

A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures

“…Our upper bound on the computational complexity depends on the diameter of the tree which is defined as (42) where is the length (number of hops) of the unique path between nodes and . For example, for the nonedge in the subtree in Fig.…”

“…Subsequent work was done in [40]- [42] albeit for differently defined uncertainty classes known as moment classes.…”

Objective information theory: A Sextuple model and 9 kinds of metrics

“…Thereafter, Efron studied the curvature of statistical manifold [46], Chentsov introduced a family of affine connections [47], and Amari put forward the concept of dual affine connections [48][49][50] which became a weaker distance function by introducing potential function to define the concept of divergence. In recent years, information geometry has been widely applied in various domains, including information theory, system theory, neural network, statistical inference, communication coding, physics and medical imaging [51][52][53][54][55][56][57][58][59][60][61][62]. As a useful tool for studying information metrics, information geometry is considered the second generation of Information Theory [63].…”

Passive dynamics in mean field control

“…See also Theorem 3.1.2 of [3] for a version of this result on a finite probability space. The papers [13], [16] contain more background and other applications of this result.…”

“…The eigenvector equation (16) implies that x Ď (x, x ) = 0 for all x ∈ X, as required for a Markovian generator. Proof of Proposition 2.2: Part (i) is essentially known: From (14) it follows that W * ∞ is the multiplicative-ergodic limit,…”