Informational analysis: a Shannon theoretic approach to measure the performance of a diagnostic test

Abstract

Diagnostic test accuracy, based on sensitivity, specificity, positive/negative predictive values (dichotomous case), and on ROC analysis (continuous case), should be expressed with a single, coherent index. We propose to modelize the diagnostic test as a flow of information between the disease, that is a hidden state of the patient, and the physicians. We assume that: i) sensitivity, specificity, false positive/negative rates are the probabilities of a Binary Asymmetric Channel ; ii) the diagnostic channel information is measured by Mutual Information. We introduce two summary measures of accuracy, namely the Information Ratio (IR) for the dichotomous case, and the Global Information Ratio (GIR) for the continuous case. We apply our model to a study by Pisano et al. [19], who compared digital versus film mammography, in diagnosing breast cancer in a screening population of 42,760 women. In film mammography, the maximum IR (0.178) corresponds to the standard cut-off of sensitivity and specificity provided by the ROC analysis (GIR 0.200). Maximum IR and GIR for digital mammography are higher (0.201 and 0.229, respectively), but IR corresponds to a cut-off with higher sensitivity but lower specificity, thus suggesting that larger information provided by digital mammography carries the risk of more false positive cases.

Fifty years of Shannon information theory in assessing the accuracy and agreement of diagnostic tests

“…Subsequently, Girometti and Fabris [ 32 ] independently developed an IT framework for diagnostic test accuracy by defining a diagnostic channel that connects the patient disease with the outcome X of the diagnostic test interpreted by the rater (see Fig. 1 ).…”

“…Based on the literature of the last fifty years, we can now state that mutual information has definitely been accepted as the correct method to measure the quantity of information extracted by a diagnostic test [ 11 – 13 , 24 , 32 , 35 , 52 , 73 ]. Concerning the medical diagnostics setting Section 2.2 , the information exchanged between D and R is measured by with the logarithm taken to the base 2.…”

Beyond kappa: an informational index for diagnostic agreement in dichotomous and multivalue ordered-categorical ratings

“…Recently, the relation between patient condition and the corresponding rater evaluation has been modelled as a virtual diagnostic information channel that transmits the exam outcomes [ 24 ]: the assumptions of a rater are based on the information obtained by the diagnostic channel, and figure out the patient condition (see Fig. 1 in [ 24 ]): the more information on the patient status flows from the patient’s real condition to the rater evaluations though the diagnostic information channel, the more accurate the diagnostic test. The channel might represent a mammography to be interpreted by a radiologist, or a prostate-specific antigen (PSA) level measure, aiding the urologist to decide whether the cut-off has been exceeded.…”

“…Recently, the relation between patient condition and the corresponding rater evaluation has been modelled as a virtual diagnostic information channel that transmits the exam outcomes [24]: the assumptions of a rater X are based on the information obtained by the diagnostic channel, Fig. 1 The agreement channel directly connects the random variables X and Y that are the terminals of the chain X ⇐⇒ diagnostic test performed by X ⇐⇒ patient condition D ⇐⇒ diagnostic test performed by Y ⇐⇒ Y and figure out the patient condition D (see Fig.…”

“…1 The agreement channel directly connects the random variables X and Y that are the terminals of the chain X ⇐⇒ diagnostic test performed by X ⇐⇒ patient condition D ⇐⇒ diagnostic test performed by Y ⇐⇒ Y and figure out the patient condition D (see Fig. 1 in [24]): the more information on the patient status flows from the patient's real condition to the rater evaluations though the diagnostic information channel, the more accurate the diagnostic test. The channel might represent a mammography to be interpreted by a radiologist, or a prostate-specific antigen (PSA) level measure, aiding the urologist to decide whether the cut-off has been exceeded.…”