How Many Participants? How Many Trials? Maximizing the Power of Reaction Time Studies

Abstract

Due to limitations in the resources available for carrying out reaction time (RT) experiments, researchers often have to choose between testing relatively few participants with relatively many trials each or testing relatively many participants with relatively few trials each. To compare the experimental power that would be obtained under each of these options, I simulated virtual experiments using subsets of participants and trials from eight large real RT datasets examining 19 experimental effects. The simulations compared designs using the first $$N_T$$ N T trials from $$N_P$$ N P randomly selected participants, holding constant the total number of trials across all participants, $$N_P \! \times \! N_T$$ N P × N T . The $$[N_P,N_T]$$ [ N P , N T ] combination maximizing the power to detect each effect depended on how the mean and variability of that effect changed with practice. For most effects, power was greater in designs having many participants with few trials each rather than the reverse, suggesting that researchers should usually try to recruit large numbers of participants for short experimental sessions. In some cases, power for a fixed total number of trials across all participants was maximized by having as few as two trials per participant in each condition. Where researchers can make plausible predictions about how their effects will change over the course of a session, they can use those predictions to increase their experimental power.

Dynamics of task preparation processes revealed by effect course analysis on response times and error rates

“…1 (for corresponding figures for ERs, see Supplementary Material C ), which also revealed no remarkable difference between raw (RT difference) and standardized effect size. This displayed effect course could recover previously reported practice-induced changes of this effect 15 : the effect initially increased until reaching its maximum between 20 and 40 trials with a maximum effect of around 100 ms, and subsequently decreased.

Figure 1 Effect course analysis of RTs for the word/nonword effect in the Adelman et al (2014) study.

…”

“…The cluster spanned from trials 13 to 323 (i.e., the end of the analyzed time window). Accordingly, effect course analysis could also recover the previously described initial lack and subsequent increase of this effect 15 , which was comparable for both the mean difference as well as the standardized effect size.

Figure 2 Effect course analysis of RTs for the response repetition effect in the Adelman et al (2014) study.

…”

“…Participants had to classify targets in a lexical decision task as word or nonword, which were preceded by a masked prime. We chose to re-analyze the word/nonword effect as well as a response repetition effect according to their previously described different time courses 15 . The word/nonword effect (slower responses for nonword than word targets) was large and present from the beginning of the experiment and initially increased further until it reached its maximum around approximately 40–50 trials and then started to decrease again.…”

“…To summarize, the aim of the present study is to apply a new method for analyzing changes of experimental effects during the course of one experimental session to investigate the plasticity of task cue effects on masked priming. In Study 1, we firstly validated the proposed effect course analysis on data of the mega-study of Adelman and colleagues 46 , for which qualitative different time courses for two effects were already described 15 . A word/nonword effect was present during the whole session, but initially increased until it reached its maximum and further started to decrease.…”

“…Such changes in cognitive processing with practice can be subsumed under the account of the plasticity of the cognitive system, which can be described as an ongoing modification of cognitive processes as a function of task practice 2 , 14 . Such a plasticity of cognitive processing is widely accepted regarding task processing, for example reflected by an overall decrease of response times during an experiment, or more specifically, by a facilitated switching between tasks with more practice 2 , 15 . However, concerning the plasticity of task preparation processes, research is lacking.…”