Small is beautiful: In defense of the small-N design

Abstract

The dominant paradigm for inference in psychology is a null-hypothesis significance testing one. Recently, the foundations of this paradigm have been shaken by several notable replication failures. One recommendation to remedy the replication crisis is to collect larger samples of participants. We argue that this recommendation misses a critical point, which is that increasing sample size will not remedy psychology’s lack of strong measurement, lack of strong theories and models, and lack of effective experimental control over error variance. In contrast, there is a long history of research in psychology employing small-N designs that treats the individual participant as the replication unit, which addresses each of these failings, and which produces results that are robust and readily replicated. We illustrate the properties of small-N and large-N designs using a simulated paradigm investigating the stage structure of response times. Our simulations highlight the high power and inferential validity of the small-N design, in contrast to the lower power and inferential indeterminacy of the large-N design. We argue that, if psychology is to be a mature quantitative science, then its primary theoretical aim should be to investigate systematic, functional relationships as they are manifested at the individual participant level and that, wherever possible, it should use methods that are optimized to identify relationships of this kind.

Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990-2012) and of latest practices (2017-2018) in high-impact journals

“…A further point to discuss regards the question of whether high population level power is always necessary for studies. We agree that small N designs may be more appropriate than large samples in some situations (Smith and Little, 2018). For example, if we study some relatively straightforward to localize motor or perceptual processes with already well-known brain anatomy and probably modest individual anatomical variability then using very high trial numbers in a few participants and fitting models to high volumes of data may be a more productive approach than testing large populations with a few trials.…”

Social anxiety modulates visual exploration in real life – but not in the laboratory

“…To the best of our knowledge, such consistency in viewing patterns has not yet been documented in a naturalistic situation outside the laboratory but it confirms that the current findings were not driven by a small number of exceptional samples in individual recordings. Crucially, extensive measurements of stable response patterns were argued to partially provide self‐replication within a single experiment and can explain why certain fields relying on experiments with small sample sizes never faced a replication crisis (Smith & Little, ). Nonetheless, while high intra‐personal stability does highlight the robustness of observations made within the present sample, future research will need to test a different group of individuals to better estimate the generalizability of between‐subject effects.…”

A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time

“…Especially questions associated with hypothesized maps of sensory representations require the sampling of large feature spaces. Therefore, large single-participant fMRI datasets seem a good addition to group studies, as it can be more useful to have a dense recording of the various stages in sensory processing in a single healthy brain rather than shorter and lower-SNR recordings of many participants [22,21,24].…”