Toward stronger tests of rationality claims: Spotlight on the rule of succession.

Abstract

A distinct statistical inference rule, the Laplacian rule of succession (RoS), is used to illustrate a general problem of decision research, namely, empirical tests of human rationality. The RoS relates the probability p of a dominant outcome in a population of binary events to an observed proportion P of the dominant outcome in a sample. The inferred probability pis generally regressive; it deviates more from the sample proportion P when samples are small rather than large. Based on computer simulations of prelated to P under various boundary conditions (measurement error; prior odds; payoff expectation, population inference vs. sample estimation), we examined judges' RoS sensitivity in a series of simple lottery experiments. Although comparative ratings of lotteries experienced in small and large samples seemed to reflect some intuitive understanding under most simplifying task conditions, they violated normative principles when complicating conditions (e.g., loss aversion, uncertainty aversion, ratio biases, Bayesian priors) overshadowed the RoS. The final discussion revolves around the difficulty, or impossibility, to conduct empirical tests of rationality.

Clarifying what reflections on rationality we intended to instigate—A reply to comments by Mandel (2022) and Costello (2022).

“…No doubt, our article on the Laplacian rule of succession (RoS) as a benchmark of rationality (Fiedler et al, 2022), which is the target of thoughtful comments by our peer reviewers David Mandel (Mandel, 2022) and Fintan Costello (Costello & Watts, 2022), cannot be reduced to a single convergent speech act. It constitutes an attempt to delineate and simulate the RoS and its diverse implications; to devise a straightforward lottery task for experimental RoS research; to document the obtained experimental results; and to raise a discussion of rival influences with which the RoS has to compete in real life.…”

Seeing evidence for rationality: Commentary on Fiedler et al. (2022).

“…Following a series of studies investigating the rationality of people’s probabilistic inference from samples, Fiedler et al (2022) reach a pessimistic conclusion: That rationality is “an idealistic criterion […] which is unlikely to be ever reached by even the smartest individuals, groups, and organizations.” Our interpretation of Fiedler et al’s results is more optimistic: We feel that they have presented new and surprising evidence for rationality in people’s inference from samples, without giving convincing evidence for irrationality in that setting.…”

“…When making judgments of association between features and categories based on samples, people reliably associate rare features with the rare category and frequent features with the frequent category (given that those features occurred at the same proportion in both categories in the sample, and no further information). Researchers typically see this as irrational, referring to it as “illusory correlation.” In an article (Costello & Watts, 2019) to which Fiedler et al (2022) is in some ways a response, we pointed out that the rational estimate for expected population probability, given a sample of size n with k feature occurrences (and no further information) is not the proportion P = k/n , but (Laplace’s Rule of Succession [RoS]). This expression is regressive toward 0.5 relative to P , with stronger regression for smaller samples.…”

“…Fiedler et al (2022) tested people’s ability to rationally follow the RoS in various ways, concluding that such rationality is “an idealistic criterion […] which is unlikely to be ever reached.” This follows from Fiedler et al’s definition of rationality as “thoughtful cognitive inferences” where “(a) the mental model is stated transparently and can thus be validated critically and (b) inferred problem solutions follow from a deliberate set of clearly articulated rules.” Under this definition, rationality becomes subjective: Even optimal inferences can be seen as irrational if we feel that the reasoner’s mental model has not been stated transparently enough, or the rules articulated clearly enough, to meet our standards. Here, we assess Fiedler et al’s (2022) results from a perspective where judgments are seen as rational when they agree with the requirements of probability theory, and irrational when they deviate from those requirements in a way that does not simply reflect random variation or noise.…”

“…Researchers typically see this as irrational, referring to it as "illusory correlation." In an article (Costello & Watts, 2019) to which Fiedler et al (2022) is in some ways a response, we pointed out that the rational estimate for expected population probability, given a sample of size n with k feature occurrences (and no further information) is not the proportion Succession [RoS]). This expression is regressive toward 0.5 relative to P, with stronger regression for smaller samples.…”