Learned Practical Guidelines for Evaluating Conditional Entropy and Mutual Information in Discovering Major Factors of Response-vs.-Covariate Dynamics

Abstract

We reformulate and reframe a series of increasingly complex parametric statistical topics into a framework of response-vs.-covariate (Re-Co) dynamics that is described without any explicit functional structures. Then we resolve these topics’ data analysis tasks by discovering major factors underlying such Re-Co dynamics by only making use of data’s categorical nature. The major factor selection protocol at the heart of Categorical Exploratory Data Analysis (CEDA) paradigm is illustrated and carried out by employing Shannon’s conditional entropy (CE) and mutual information (I[Re;Co]) as the two key Information Theoretical measurements. Through the process of evaluating these two entropy-based measurements and resolving statistical tasks, we acquire several computational guidelines for carrying out the major factor selection protocol in a do-and-learn fashion. Specifically, practical guidelines are established for evaluating CE and I[Re;Co] in accordance with the criterion called [C1:confirmable]. Following the [C1:confirmable] criterion, we make no attempts on acquiring consistent estimations of these theoretical information measurements. All evaluations are carried out on a contingency table platform, upon which the practical guidelines also provide ways of lessening the effects of the curse of dimensionality. We explicitly carry out six examples of Re-Co dynamics, within each of which, several widely extended scenarios are also explored and discussed.

Unraveling implicit human behavioral effects on dynamic characteristics of Covid-19 daily infection rates in Taiwan

“…In the second subsection, we provide a concise overview of the major factor selection protocol used to examine the dynamics of a chosen response variable with respect to a set of covariate features. This protocol, developed in [12][13][14][15], builds upon the concepts of conditional entropy and mutual information.…”

“…Major factors refer to a feature set that explains a significant proportion of the uncertainty in the response variable. This data-driven computational protocol, based on Theoretical Information Measurements such as conditional entropy and mutual information [ 11 ], has been developed in previous works [ 12 – 15 ]. We will provide a brief introduction to these key ideas in Section 4.…”

Unraveling Heterogeneity of ADNI’s Time-to-Event Data Using Conditional Entropy—Part I: Cross-Sectional Study

“…This paper proposes the Categorical Exploratory Data Analysis (CEDA) paradigm to algorithmically resolve all issues raised from Q1 through Q5 and partially Q6 and Q7 [11], [12], [13], [14]. In CEDA, we apply Shannon conditional entropy to evaluate associative relation, and mutual information to select so-called major factors underlying the dynamics of the response variable in relation to all covariate feature variables.…”

“…In this subsection, we briefly review the concept and computing of conditional entropy and mutual information as two key Theoretical Information Measurements used throughout this paper. Detailed derivations of related formulas of these two measurements are referred to previous works [11], [12], [13], [14]. We employ these two entropy-based measurements to evaluate potentially nonlinear directional association from a generic covariate feature variable denoted as X to a generic response variable denoted as Y, and the nondirectional association between two covariate feature variables, say X 1 and X 2 .…”