Don’t Forget the Psychology in Analyses of Psychological Data: The Case of Sequential Testing

Abstract

Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results. Although there are approaches that can mitigate many statistical issues with sequential testing, we suggest that current discussions of the topic are limited by focusing almost entirely on the mathematical underpinnings of analytic approaches. An important but largely neglected assumption of sequential testing is that the data generating process under investigation remains constant across the experimental cycle. Without care, psychological factors may result in violations of this assumption when sequential testing is used: researchers’ behavior may be changed by the observation of incoming data, in turn influencing the process under investigation. We argue for the consideration of an ‘insulated’ sequential testing approach, in which research personnel remain blind to the results of interim analyses. We discuss different ways of achieving this, from automation to collaborative inter-lab approaches. As a practical supplement to the issues we raise, we introduce an evolving resource aimed at helping researchers navigate both the statistical and psychological pitfalls of sequential testing: the Sequential Testing Hub (www.sequentialtesting.com). The site includes a guide for involving an independent analyst in a sequential testing pipeline, an annotated bibliography of relevant articles covering statistical aspects of sequential testing, links to tools and tutorials centered around how to actually implement a sequential analysis in practice, and space for suggestions to help develop this resource further. We aim to show that although unfettered use of sequential testing may raise problems, carefully designed procedures can limit the pitfalls arising from its use, allowing researchers to capitalize on the benefits it provides.

Demarcating the boundary conditions of memory reconsolidation: An unsuccessful replication

“…Interim analyses were performed by a researcher (LS) who was not directly involved with participants to prevent changes in experimenter behaviour based on awareness of the direction of the effect 25 . For the results of the interim analyses, see the supplementary material S1 .…”

“…We aimed to test a minimum of 60 participants (20 per group, which is comparable to the original sample), and for practical reasons we a priori decided to stop at a maximum of 90 participants, while performing intermittent checks every 6 participants. Importantly, the researcher collecting the data was not aware of the results of the interim analyses to avoid potential effects on experimenter behaviour 25 .…”

Over the Edge: Extending the duration of a reconsolidation intervention for spider fear

“…Figure 1 shows all in-session exclusion criteria and the number of excluded participants. The insulated sequential analysis [ 34 ] had three stopping criteria; (1) a one-sided Bayesian Mann–Whitney U test indicated a Bayes Factor ≥10 for or against the prediction that tarantula behavioural approach task approach scores would be higher following the brief reactivation procedure than the extended; (2) less than 50% of participants in the brief condition touched the tarantula after treatment; (3) the maximum sample size of 50 was reached. Data checks were performed after the first 20 participants, then after each 5 (within 1 of equally balanced across groups).…”

“…AF conducted all treatment sessions (t1) and was only unblinded upon entering the reactivation procedure room to ensure consistency across groups for as long as possible. An independent researcher (JE) conducted the insulated sequential analyses [ 34 ], keeping authors and research personnel blind to interim results, beyond whether to continue collecting data.…”

The Registration Continuum in Personality Disorder Studies: Theory, Rationale, and Template

“…Registered flexibility also allows for principled control of error while studies progress. For instance, sequential analyses allow researchers to examine their data at prespecified intervals in recruitment or a study's procedures, permitting an efficient use of scarce participants and limited resources when the interim analyses are masked against experimenter expectancies (Elsey et al, 2021).…”