An Optimization Approach of Deriving Bounds between Entropy and Error from Joint Distribution: Case Study for Binary Classifications

Abstract

Abstract:In this work, we propose a new approach of deriving the bounds between entropy and error from a joint distribution through an optimization means. The specific case study is given on binary classifications. Two basic types of classification errors are investigated, namely, the Bayesian and non-Bayesian errors. The consideration of non-Bayesian errors is due to the facts that most classifiers result in non-Bayesian solutions. For both types of errors, we derive the closed-form relations between each bound and error components. When Fano's lower bound in a diagram of "Error Probability vs. Conditional Entropy" is realized based on the approach, its interpretations are enlarged by including non-Bayesian errors and the two situations along with independent properties of the variables. A new upper bound for the Bayesian error is derived with respect to the minimum prior probability, which is generally tighter than Kovalevskij's upper bound.

X-ray measurement model incorporating energy-correlated material variability and its application in information-theoretic system analysis

“…where An upper bound on binary classification errors P e , which is tighter than Kovalevskij's inequality, has been reported recently [13],…”

“…Shannon mutual information, I S , has long been used as a metric to quantify the task-specific fidelity of a measurement with respect to classification tasks [8]. This is because I S is related to the error probability, P e through Fano's inequality [10] and Kovalevskij's inequality [11][12][13].…”

“…The difference in bound on P e calculated from the energy-uncorrelated model and the energycorrelated model is due to the difference in the material variation considered in the two models, as indicated by the difference in the red-dashed lines. When energy correlation in the data is considered, Σ J , which is the covariance matrix of J(E), is calculated based on Equation (13); in contrary, when energy correlation is not considered, the off-diagonal elements of Σ J are ignored. Since the energy-binning process is an integration over energy, each element of Σ J d is the sum of a block matrix in Σ J .…”

“…Task-specific information (TSI) [8], which is the information content relevant to the task, is commonly used as an objective assessment metric. For classification tasks, Shannon mutual information (I S ) [9] is a natural choice as an information-theoretic metric, because it can be used to bound probability of classification error (P e ) [9][10][11][12][13]. Although I S is expensive to compute for non-trivial distributions, we are able to derive closed-form expressions for bounds on I S and bounds on P e for mixture distributions with the help of a recent work [14].…”

Information Theory and Its Relation to Machine Learning

“…The proof is neglected in this paper, but it can be given based on the study of bounds between entropy and error (cf. [27] and references therein). The significance of Theorem 1 implies that an optimization of information measure may not guarantee to achieve an optimization of the empirically-defined similarity measure.…”