Better define beta–optimizing MDD (minimum detectable difference) when interpreting treatment-related effects of pesticides in semi-field and field studies

Abstract

The minimum detectable difference (MDD) is a measure of the difference between the means of a treatment and the control that must exist to detect a statistically significant effect. It is a measure at a defined level of probability and a given variability of the data. It provides an indication for the robustness of statistically derived effect thresholds such as the lowest observed effect concentration (LOEC) and the no observed effect concentration (NOEC) when interpreting treatment-related effects on a population exposed to chemicals in semi-field studies (e.g., micro-/mesocosm studies) or field studies. MDD has been proposed in the guidance on tiered risk assessment for plant protection products in edge of field surface waters (EFSA Journal 11(7):3290, 2013), in order to better estimate the robustness of endpoints from such studies for taking regulatory decisions. However, the MDD calculation method as suggested in this framework does not clearly specify the power which is represented by the beta-value (i.e., the level of probability of type II error). This has implications for the interpretation of experimental results, i.e., the derivation of robust effect values and their use in risk assessment of PPPs. In this paper, different methods of MDD calculations are investigated, with an emphasis on their pre-defined levels of type II error-probability. Furthermore, a modification is suggested for an optimal use of the MDD, which ensures a high degree of certainty for decision-makers.

The Minimum Detectable Difference (MDD) Concept for Establishing Trust in Nonsignificant Results: A Critical Review

“…The difficulty is that we want to get knowledge about the real world (real effect), but the MDD only provides information about an effect on the measurement level: for a given sample size and estimated variance, an estimated (i.e., measured) effect of the size of the MDD would be significant, always. A real effect size equal to the MDD, however, will be significant in only 50 to 60% of the conducted experiments (see The MDD Does Not Have Controlled Power section; Duquesne et al 2020), which is far below the usually targeted power of 80% (European Food Safety Authority 2013; European Food Safety Authority PPR Panel 2015). The same differentiation has to be applied to the interpretation of the MDD as an upper bound for the real effect: for a nonsignificant experiment, the estimated effect will always be smaller than the MDD (information that is already covered by the nonsignificant p value; see also Hoenig and Heisey 2001; Colegrave and Ruxton 2003).…”

“…For a one‐sided t test, the corresponding power is the area below the t ‐distribution from t ‐critical (which corresponds to the MDD; see Figure 3) to infinity (red area below the solid line in Figure 4A). If the t ‐distribution for a real effect of size MDD (i.e., around t ‐critical) was symmetric (which is true asymptotically), the corresponding power would be 50%, because the values of the t ‐distribution would fall in equal proportions on either side of t ‐critical (Hoenig and Heisey 2001; Duquesne et al 2020). Thus, asymptotically, the MDD has a power of 50%, which would not be considered high according to normal statistical standards.…”

“…Given the definition of the MDD as the effect size that would have been significant, it is tempting to think that 1) there would be a high power to detect an effect of the size of the MDD (as suggested, e.g., in Andrade et al 2017; Green et al 2018) and 2) the power to detect an effect of the size of the MDD is fixed (as suggested, e.g., in Duquesne et al 2020). Neither is true.…”

“…This issue is particularly worrisome for higher tier experiments such as mesocosm studies, because these costly experiments are often conducted with low sample sizes (Touart 1994; Sanderson 2002; de Jong et al 2006; Organisation for Economic Co‐operation and Development 2007; Cabrera et al 2016; Lemoine et al 2016) and show large natural variation (Liber et al 1992; Kraufvelin 1998). Both factors contribute to low power, and thus high type II error rates (i.e., false negatives; Organisation for Economic Co‐operation and Development 2006; European Food Safety Authority PPR Panel 2013; Duquesne et al 2020). To establish a reasonable regulatory framework, it is therefore essential to complement null hypothesis significance tests with minimum requirements on the power of the experiment (SANCO 2002a; Sanderson and Petersen 2002; de Jong et al 2006; Organisation for Economic Co‐operation and Development 2006; European Food Safety Authority 2013; European Food Safety Authority PPR Panel 2015), or find some other means to identify false negatives arising from low power.…”

A two years field experiment to assess the impact of two fungicides on earthworm communities and their recovery

“…Moreover, the MDD calculation allows the assessment of risks considering the different effect classes and specific protection goals as proposed by the EFSA's opinion (EFSA PPR Panel, 2017). In our study, the MDD (Duquesne et al, 2020) of our Dunnett's tests ranged from 74,3% to 83,9%, using power calculations for t tests with R software and transposing the MDDs obtained to the Dunnett's test with 6 groups of 4 samples with a bootstrap approach. A pre-sampling before the beginning of the study is also recommended in order to get an overview of soil organism distribution in the field and to balance sampling size and statistical relevance.…”