Considering field factors to enhance external validity of scenario planning experiments: A commentary on Derbyshire et al. (2022)

The value of experiments in futures and foresight science: A reply

“…This usefully highlights the difference between a conceptual replication—or “conceptual extension” as Tsang and Kwan (1999) alternatively call it (more on that subject later)—and a direct replication. When we said that Phadnis et al (2015) “did not replicate” Schoemaker (1993), we were referring to Phadnis et al (2015) as a conceptual replication/extension of Schoemaker (1993), which it indeed was as acknowledged in Phadnis’ (2022) response to our paper. Even if Phadnis et al (2015) did not seek directly to replicate Schoemaker (1993), it was a conceptual replication/extension of it.…”

“…Indeed, Phadnis et al (2015) conducted a field experiment as well as a conceptual extension of Schoemaker (1993). Phadnis (2022) highlights the recognition in our paper of two threats to external validity—unrepresentative sample and unrepresentative design. The former is when a study's participants are not representative of the target population and the latter is when the experimental context is not representative of the real‐world decision context.…”

“…Phadnis (2022) recognizes the possibility for random assignment to treatment and control groups, which can increase what we referred to in our paper as internal validity, but also recognizes that the random sampling needed to establish the external validity of an experiment is all but impossible in an organizational context. However, Phadnis (2022) proffers a potential solution to this problem. Namely, field experiments, which are experiments conducted in the natural setting in which the phenomenon under study occurs.…”

“…Such artificially motivated scenario planning conducted simply to increase experimentation would lack external validity. It would certainly fall foul of the principles for designing field experiments highlighted by Phadnis (2022). It is for all these reasons that we emphasized the importance of representative design in our paper.…”

“…Finally, the difficulties associated with conducting experiments on scenario planning again resurface in Phadnis (2022). Phadnis (2022) recognizes the possibility for random assignment to treatment and control groups, which can increase what we referred to in our paper as internal validity, but also recognizes that the random sampling needed to establish the external validity of an experiment is all but impossible in an organizational context. However, Phadnis (2022) proffers a potential solution to this problem.…”