Kahneman and Tversky's 1979 article on Prospect Theory is one of the most influential papers across all of the behavioural sciences. The study tested a series of binary financial (risky) choices, ultimately concluding that judgments formed under uncertainty deviate significantly from those presumed by expected utility theory, which was the prevailing theoretical construct at the time. In the forty years since publication, this study has had a remarkable impact on science, policy, and other real-world applications. At the same time, a number of critiques have been raised about its conclusions and subsequent constructs that were founded on it, such as loss aversion. In an era where such presumed canonical theories have increasingly drawn scrutiny for inability to replicate, we attempted a multinational study of N = 4,099 participants from 19 countries and 13 languages. The same methods and procedures were used as in the original paper, adjusting only currencies to make them relative to current values, and requiring all participants to respond to all items. Overall, we found that results replicated for 94% of the 17 choice items tested. At most, results from the 1979 study were attenuated in our findings, which is most likely due to a more robust sample. Twelve of the 13 theoretical contrasts presented by Kahneman and Tversky also replicated, with a further 89% replication rate of the total contrasts possible when separating by location, up to 100% replication in some countries. We conclude that the principles of Prospect Theory replicate beyond any reasonable thresholds, and provide a number of important insights about replications, attenuation, and implications for the study of human decision-making at population-level.

In the past two decades, psychological science has experienced an unprecedented replicability crisis which uncovered several problematic issues. Among others, the use and misuse of statistical inference plays a key role in this crisis. Indeed, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. Instead, statistical reasoning is necessary both at the planning stage and when interpreting the results of a research project. Based on these considerations, we build on and further develop an idea proposed by Gelman and Carlin (2014) termed "prospective and retrospective design analysis".Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant), and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers awareness during all phases of a research project. To illustrate the benefits of design analysis to the widest possible audience, we use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups considering Cohens d as an effect size measure. We examine the case in which the plausible effect size is formalized as a single value, and propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples and an application to a real case study, we show that even though a design analysis requires big effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed. Keywordsprospective and retrospective design analysis, Type M and Type S errors, effect size, power, psychological research, statistical inference, statistical reasoning, R functions "If statisticians agree on one thing, it is that scientific inference should not be made mechanically." Gigerenzer and Marewski (2015, p. 422) "Accept uncertainty. Be thoughtful, open, and modest.

It is widely appreciated that many studies in psychological science suffer from low statistical power. One of the consequences of analyzing underpowered studies with thresholds of statistical significance, is a high risk of finding exaggerated effect size estimates, in the right or in the wrong direction. These inferential risks can be directly quantified in terms of Type M (magnitude) error and Type S (sign) error, which directly communicate the consequences of design choices on effect size estimation. Given a study design, Type M error is the factor by which a statistically significant effect is on average exaggerated. Type S error is the probability to find a statistically significant result in the opposite direction to the plausible one. Ideally, these errors should be considered during a prospective design analysis in the design phase of a study to determine the appropriate sample size. However, they can also be considered when evaluating studies’ results in a retrospective design analysis. In the present contribution we aim to facilitate the considerations of these errors in the research practice in psychology. For this reason we illustrate how to consider Type M and Type S errors in a design analysis using one of the most common effect size measures in psychology: Pearson correlation coefficient. We provide various examples and make the R functions freely available to enable researchers to perform design analysis for their research projects.

Kahneman and Tversky’s 1979 article on Prospect Theory is one of the most influential papers across all of the behavioural sciences. The study tested a series of binary financial (risky) choices, ultimately concluding that judgments formed under uncertainty deviate significantly from those presumed by expected utility theory, which was the prevailing theoretical construct at the time. In the forty years since publication, this study has had a remarkable impact on science, policy, and other real-world applications. At the same time, a number of critiques have been raised about its conclusions and subsequent constructs that were founded on it, such as loss aversion. In an era where such presumed canonical theories have increasingly drawn scrutiny for inability to replicate, we attempted a multinational study of N = 4,099 participants from 19 countries and 13 languages. The same methods and procedures were used as in the original paper, adjusting only currencies to make them relative to current values, and requiring all participants to respond to all items. Overall, we found that results replicated for 94% of the 17 choice items tested. At most, results from the 1979 study were attenuated in our findings, which is most likely due to a more robust sample. Twelve of the 13 theoretical contrasts presented by Kahneman and Tversky also replicated, with a further 89% replication rate of the total contrasts possible when separating by location, up to 100% replication in some countries. We conclude that the principles of Prospect Theory replicate beyond any reasonable thresholds, and provide a number of important insights about replications, attenuation, and implications for the study of human decision-making at population-level.

Design Analysis was introduced by Gelman & Carlin (2014) as an extension of Power Analysis. Traditional power analysis has a narrow focus on statistical significance. Design analysis, instead, evaluates together with power levels also other inferential risks (i.e., Type M error and Type S error), to assess estimates uncertainty under hypothetical replications of a study.Given an hypothetical value of effect size and study characteristics (i.e., sample size, statistical test directionality, significance level), Type M error (Magnitude, also known as Exaggeration Ratio) indicates the factor by which a statistically significant effect is on average exaggerated. Type S error (Sign), instead, indicates the probability of finding a statistically significant result in the opposite direction to the hypothetical effect.Although Type M error and Type S error depend directly on power level, they underline valuable information regarding estimates uncertainty that would otherwise be overlooked. This enhances researchers awareness about the inferential risks related to their studies and helps them in the interpretation of their results. However, design analysis is rarely applied in real research settings also for the lack of dedicated software.To know more about design analysis consider Gelman & Carlin (2014) and Lu et al. (2018). While, for an introduction to design analysis with examples in psychology see Altoè et al. (2020) andBertoldo et al. (2020).