Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

Abstract

Several norms for how people should assess a question's usefulness have been proposed, notably Bayesian diagnosticity, information gain (mutual information), Kullback-Liebler distance, probability gain (error minimization), and impact (absolute change). Several probabilistic models of previous experiments on categorization, covariation assessment, medical diagnosis, and the selection task are shown to not discriminate among these norms as descriptive models of human intuitions and behavior. Computational optimization found situations in which information gain, probability gain, and impact strongly contradict Bayesian diagnosticity. In these situations, diagnosticity's claims are normatively inferior. Results of a new experiment strongly contradict the predictions of Bayesian diagnosticity. Normative theoretical concerns also argue against use of diagnosticity. It is concluded that Bayesian diagnosticity is normatively flawed and empirically unjustified.

Missing the dog that failed to bark in the nighttime: on the overestimation of occurrences over non-occurrences in hypothesis testing

“…However, several alternative (Bayesian) models of the utility value of clues have been proposed (e.g., Crupi, Tentori, & Gonzalez, 2007;Nelson, 2005Nelson, , 2008. In particular, recent experimental work by Nelson, McKenzie, Cottrell, and Sejnowski (2010) has shown that probability gain predicted human information search better than other measures of the value of information (but see Nelson 2005 for data showing that information gain and KullbackLeibler distance were slightly better predictors than probability gain and impact). According to probability gain the information value of the presence of a feature (e.g., the "yes" answer to a dichotomous question) is computed as:…”

“…In tasks in which the prior probabilities of the hypotheses are equiprobable, probability gain and impact lead to the same values of information of a datum while information gain makes the same predictions of the Kullback-Leibler distance (e.g., Nelson 2005Nelson , 2008Nelson et al, 2010).…”

Evidence evaluation: Measure Z corresponds to human utility judgments better than measure L and optimal-experimental-design models.

“…Among them are two inductive confirmation measures, namely, measures L and Z (e.g., Crupi, Tentori, & Gonzalez, 2007;Fitelson, 2001Fitelson, , 2006Mastropasqua, Crupi, & Tentori, 2010;Tentori, Crupi, Bonini, & Osherson, 2007), and OED models (Nelson, 2005(Nelson, , 2008(Nelson, , 2009Nelson et al, 2010), namely, Bayesian diagnosticity, log 10 diagnosticity, information gain, Kullback-Leibler distance, probability gain, and impact. Although they differ in terms of how the utility of the obtained evidence is calculated, both of these classes of normative models are based on Bayes' rule, and thus, they involve prior probabilities (which express one's initial beliefs with respect to one or more hypotheses), likelihoods (which indicate the probability of the new evidence), and posterior probabilities (which express one's beliefs in light of the new evidence).…”

“…Furthermore, different normative standards, often called optimal-experimental-design (OED) models, have been proposed as models of human intuition about the value of information (e.g., Meder & Nelson, 2012;Nelson, 2005Nelson, , 2008Nelson, McKenzie, Cottrell, & Sejnowski, 2010). However, OED models have been investigated only with respect to their adequacy in describing human information gathering (e.g., Meder & Nelson, 2012;Nelson, 2005Nelson, , 2008Nelson et al, 2010).…”

“…However, OED models have been investigated only with respect to their adequacy in describing human information gathering (e.g., Meder & Nelson, 2012;Nelson, 2005Nelson, , 2008Nelson et al, 2010). Therefore, further empirical investigation is needed to examine their relative descriptive power compared with measures L and Z as well as their absolute ability to capture human intuition in evidence evaluation, for example, when physicians scrutinize test outcomes, MODELS OF EVIDENCE EVALUATION 5 when scientists interpret experiment results or, more generally, when one receives a "yes" or a "no" answer to a question.…”