Assessing Effects of Unreplicated Perturbations: No Simple Solutions

Abstract

We address the task of determining the effects, on mean population density or other parameters, of an unreplicated perturbation, such as arises in environmental assessments and some ecosystem—level experiments. Our context is the Before—After—Control—Impact—Pairs design (BACIP): on several dates Before and After the perturbation, samples are collected simultaneously at both the Impact site and a nearby "Control." One approach is to test whether the mean of the Impact—Control difference has changed from Before to After the perturbation. If a conventional test is used, checks of its assumptions are an important and messy part of the analysis, since BACIP data do not necessarily satisfy them. It has been suggested that these checks are not needed for randomization tests, because they are insensitive to some of these assumptions and can be adjusted to allow for others. A major aim of this paper is to refute this suggestion: there is no panacea for the difficult and messy technical problems in the analysis of data from assessments or unreplicated experiments. We compare the randomization t test with the standard t test and the modified (Welch—Satterthwaite—Aspin) t test, which allows for unequal variances. We conclude that the randomization t test is less likely to yield valid inferences than is the Welch t test, because it requires identical distributions for small sample sizes and either equal variances or equal sample sizes for larger ones. The formal requirement of Normality is not crucial to the Welch t test. Both parametric and randomization tests require that time and location effects be additive and the Impact—Control differences on different dates be independent. These assumptions should be tested; if they are seriously wrong, alternative analyses are needed. This will often require a long time series of data. Finally, for assessing the importance of a perturbation, the P value of a hypothesis test is rarely as useful as an estimate of the size of the effect. Especially if effect size varies with time and conditions, flexible estimation methods with approximate answers are preferable to formally exact P values.

Influence of annual variation in food supply on abundance of wintering common eiders Somateria mollissima

“…Therefore, we used an alpha level of 0.01 to calculate confidence intervals. With the sample sizes and the level of autocorrelation found by Guillemette et al, an alpha level of 0.01 corresponds approximately to a real alpha level of 0.10 (see Stewart-Oaten et al 1992). Differences between years were inferred when the mean of numbers in one year was not included in the confidence interval of another year.…”

Artificial Light at Night Affects Organism Flux across Ecosystem Boundaries and Drives Community Structure in the Recipient Ecosystem

“…An un-replicated BACI design such as this assesses how a system reacts to perturbation in comparison to a simultaneously studied unperturbed control system (Stewart-Oaten et al, 1992). In our case, the problem of un-replicated perturbation was addressed in four ways: (i) an effect of ALAN is indicated by a significant interaction of the factor "site" (i.e., Control/Treatment) with "phase" (i.e., before/after perturbation) rather than by a direct difference (i.e., main effect) between the two sites; (ii) mean sample values at each site at any point in time are reliably estimated by extensive replication using multiple traps; (iii) multiple sampling occasions in the before and after phases provide reliable means for both phases and the necessary degrees of freedom for the statistical tests; and (iv) potential confounding effects are excluded by striving for similarity in environmental conditions (other than those directly related to ALAN) across sites and assessing stability of this similarity (or persistence of any dissimilarity) between phases.…”

A Long-Term Study of Ecological Impacts of River Channelization on the Population of an Endangered Fish: Lessons Learned for Assessment and Restoration

“…Although widely applied to environmental assessments, application of the BACI framework within a riverine system required us to make several simplistic and untenable assumptions (e.g., impacts were applied suddenly and uniformly to the I reach, C and I reaches were demographically independent, and environmental factors other than the RRFRP were constant; Table 2) that are probably unmet in many other BACI applications or in other analyses aimed at assessing effects of unreplicated alterations [99]. These assumptions can lead to biased conclusions that may be vetted by adopting alternative analytical frameworks or testing for impacts on multiple ecological indicators (e.g., adults and juveniles).…”