Permutation tests for experimental data

Abstract

This article surveys the use of nonparametric permutation tests for analyzing experimental data. The permutation approach, which involves randomizing or permuting features of the observed data, is a flexible way to draw statistical inferences in common experimental settings. It is particularly valuable when few independent observations are available, a frequent occurrence in controlled experiments in economics and other social sciences. The permutation method constitutes a comprehensive approach to statistical inference. In two-treatment testing, permutation concepts underlie popular rank-based tests, like the Wilcoxon and Mann–Whitney tests. But permutation reasoning is not limited to ordinal contexts. Analogous tests can be constructed from the permutation of measured observations—as opposed to rank-transformed observations—and we argue that these tests should often be preferred. Permutation tests can also be used with multiple treatments, with ordered hypothesized effects, and with complex data-structures, such as hypothesis testing in the presence of nuisance variables. Drawing examples from the experimental economics literature, we illustrate how permutation testing solves common challenges. Our aim is to help experimenters move beyond the handful of overused tests in play today and to instead see permutation testing as a flexible framework for statistical inference.

Multivariate links between the developmental timing of adversity exposure and white matter tract integrity in adulthood

“…This time-intensive phenotyping is critical to advancing the science of adversity exposure and brain development, yet practically constrains that the sample size we could feasibly collect. While we aimed to characterize our sample as robustly as possible by applying resampling and permutation testing 46 and utilizing a multivariate approach 47 , it will be important to replicate these findings in a larger, more demographically diverse sample in order to evaluate robustness and generalizability. Further, despite having rich information on adversity exposure, sample size precluded us from further parsing these events by additional relevant dimensions of adversity, such as perceived severity, threat, deprivation, controllability, and predictability.…”

Spatial and Temporal Quality of Brain Networks for Different Multi-Echo fMRI Combination Methods

“…For all t-tests we also ran separate permutation tests to evaluate the validity oof the paired ttests. Permutation tests could be more accurate because there are no distributional assumptions that have to be met and they are an exact approximation of the type I error [38], [39]. We ran a paired sample permutation test based on a t-statistic [40] using a Matlab package [41] and corrected p-values again with Holm-Bonferroni testing.…”

Evaluating the Electroencephalographic Signal Quality of an In-Ear Wearable Device

“…In order to test for the statistical significance of the linear correlation coefficients obtained between the in-ear and scalp EEG signals, we implemented a Pitman nonparametric permutation test [ 35 , 36 ]. Nonparametric statistical testing has been widely used in neuroimaging studies [ 37 , 38 ].…”

“…As recommended [ 35 ], we repeated this procedure 1600 times (permutations) to generate a distribution of 1600 coefficient values from the corresponding permutations. We then estimated the two-sided p -value as the proportion of the absolute values of correlation coefficients from shuffled permutations that were larger than the true absolute value of the Pearson correlation coefficient calculated from the original data series [ 36 ]. We defined a statistical significance level for p < 0.05.…”