Power contours: Optimising sample size and precision in experimental psychology and human neuroscience.

Baker, Daniel H.; Vilidaitė, Greta; Lygo, Freya; Smith, Anika; Flack, Tessa R.; Gouws, André; Andrews, Timothy J.

doi:10.1037/met0000337

Cited by 190 publications

(170 citation statements)

References 47 publications

Supporting

Mentioning

163

Contrasting

Unclassified

Order By: Relevance

“…Increasing the number of repetitions per condition can be applied for all diagnostic assessment procedures with homogeneous items. The role of the number of trials in experimental psychology tasks has been recently raised as an important factor to be considered to ensure sufficient power (Baker et al, 2019;Brysbaert, 2019). When more trials are included for averaging, the errors associated with the single measurements tend to cancel each other out more accurately 16 .…”

Section: Improving Snarc Effect Reliability and Precisionmentioning

confidence: 99%

A Minority pulls the sample mean: on the individual prevalence of robust group-level cognitive phenomena – the instance of the SNARC effect

Cipora

Dijck

Georges

et al. 2019

Preprint

View full text Add to dashboard Cite

The aim of cognitive psychology is to obtain insights into human cognition in general. For this purpose, group-studies are typically conducted on representative samples so that the results can be generalized to the population. Using this approach, individual differences in such group-level cognitive phenomena are typically neglected and not much is known about their prevalence at the individual level. Such information is nevertheless important for claims about the universality of phenomena, as in theory, significant effects at the group-level can in principle be driven by a minority of participants. Here we used a uniform analysis of 18 existing data sets revealing a well-replicated phenomenon in numerical cognition: the SNARC (Spatial-Numerical Association of Response Codes) effect, in order to investigate the prevalence of the effect at the individual level. Three methods of analyzing the presence of the effect at the individual level were utilized: one psychometric and two bootstrapping methods. The results show that the group-level SNARC effect is driven by a minority of individuals (≤ 45%) who reveal the effect. This finding demonstrate an important theoretical issue: whether group-level effects really reflect general principles of cognition. We discuss advantages and drawbacks of the present methods and their usefulness for investigating the prevalence of other cognitive phenomena. We posit that testing the presence of robust group-level cognitive effects at the individual level as well as ensuring their reliable measurement is an important step towards integrating two traditionally separate approaches of scientific psychology proposed back in 50’ by Cronbach: experimental and correlational.

show abstract

Section: Improving Snarc Effect Reliability and Precisionmentioning

confidence: 99%

A Minority pulls the sample mean: on the individual prevalence of robust group-level cognitive phenomena – the instance of the SNARC effect

Cipora

Dijck

Georges

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…From this perspective, power should be calculated for effects within individual participants. This gives a very different view of the strength of evidence provided by a data set and of the importance of sample size (for both participants and trials) compared to the more common population mean perspective (Baker et al, 2019). For example, the simulation of 50 participants with 20 trials in Figure 1C has = 0.008 for a group mean different from zero, a result that is as surprising, under the null hypothesis, as observing 7 heads in a row from tosses of a fair coin (Obleser, 2019).…”

Section: Discussionmentioning

confidence: 99%

“…Figure 2A-C suggests that for the Bayesian prevalence metrics, there are benefits to having larger numbers of participants (decrease in variance of obtained MAP and HPDI width, increase in prevalence lower bound), but beyond around 50 participants these benefits become less pronounced. Figures 2E shows that inferred prevalence is mostly sensitive to the number of trials per participant (horizontal contours), and invariant to the number of participants (although variance increases as in Figure 2A,C,F), whereas t-test power ( Figure 2D) is mostly sensitive to number of participants (vertical contours) and largely invariant to number of trials beyond around 100 trials per participant (Baker et al, 2019). In sum, compared to the population mean t-test, prevalence exhibits greater sensitivity to the number of trials obtained per participant, and less sensitivity to the number of participants.…”

Section: Estimating Population Prevalencementioning

confidence: 93%

Bayesian inference of population prevalence

Ince

Kay

Schyns

2020

Preprint

View full text Add to dashboard Cite

AbstractWithin neuroscience, psychology and neuroimaging, the most frequently used statistical approach is null-hypothesis significance testing (NHST) of the population mean. An interesting alternative is to perform NHST within individual participants and then infer, from the proportion of participants showing an effect, the prevalence of that effect in the population. We propose a novel Bayesian method to estimate such population prevalence which has several advantages over population mean NHST. First, it provides a population level inference currently missing for designs with small numbers of participants such as traditional psychophysics, animal electrophysiology and precision imaging. Second, it delivers a quantitative estimate with associated uncertainty instead of reducing an experiment to a binary inference on a population mean. Bayesian prevalence is widely applicable to a broad range of studies in neuroscience, psychology and neuroimaging. Its emphasis on detecting effects within individual participants could help address replicability issues. To facilitate the applicability of Bayesian prevalence, we provide code in Matlab, Python and R.

show abstract

“…Their focus is on comparing between different decision models rather than model-free and model-based transformations of reaction time and accuracy. Baker et al (2019) used the simulation method to address a question of equal importance to experimentalists-how does the number of trials interact with sample size to affect statistical power? Like us, they present an interactive demonstration of their findings https://shiny.york.ac.uk/ powercontours/…”

Section: Related Workmentioning

confidence: 99%

Quantifying the benefits of using decision models with response time and accuracy data

et al. 2020

View full text Add to dashboard Cite

Response time and accuracy are fundamental measures of behavioral science, but discerning participants' underlying abilities can be masked by speed-accuracy trade-offs (SATOs). SATOs are often inadequately addressed in experiment analyses which focus on a single variable or which involve a suboptimal analytic correction. Models of decision-making, such as the drift diffusion model (DDM), provide a principled account of the decision-making process, allowing the recovery of SATO-unconfounded decision parameters from observed behavioral variables. For plausible parameters of a typical between-groups experiment, we simulate experimental data, for both real and null group differences in participants' ability to discriminate stimuli (represented by differences in the drift rate parameter of the DDM used to generate the simulated data), for both systematic and null SATOs. We then use the DDM to fit the generated data. This allows the direct comparison of the specificity and sensitivity for testing of group differences of different measures (accuracy, reaction time, and the drift rate from the model fitting). Our purpose here is not to make a theoretical innovation in decision modeling, but to use established decision models to demonstrate and quantify the benefits of decision modeling for experimentalists. We show, in terms of reduction of required sample size, how decision modeling can allow dramatically more efficient data collection for set statistical power; we confirm and depict the non-linear speed-accuracy relation; and we show how accuracy can be a more sensitive measure than response time given decision parameters which reasonably reflect a typical experiment.

show abstract

Power contours: Optimising sample size and precision in experimental psychology and human neuroscience.

Cited by 190 publications

References 47 publications

A Minority pulls the sample mean: on the individual prevalence of robust group-level cognitive phenomena – the instance of the SNARC effect

A Minority pulls the sample mean: on the individual prevalence of robust group-level cognitive phenomena – the instance of the SNARC effect

Bayesian inference of population prevalence

Quantifying the benefits of using decision models with response time and accuracy data

Contact Info

Product

Resources

About