Robust Permutation Tests For Correlation And Regression Coefficients

DiCiccio, Cyrus; Romano, Joseph P.

doi:10.1080/01621459.2016.1202117

Cited by 77 publications

(125 citation statements)

References 15 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…Until recently, the test was not popular due to high computational demands (Hayes, 1996;Ludbrook & Dudley, 1998). Since the availability of fast computers, permutation tests have been developed for correlation and regression (Anderson & Robinson, 2001;DiCiccio & Romano, 2017), ANOVA (Anderson, 2001), the general linear model (Winkler, Ridgway, Webster, Smith, & Nichols, 2014), and qualitative comparative analysis (Braumoeller, 2015). The permutation test produces a p value.…”

Section: Permutation Testmentioning

confidence: 99%

A Statistical Significance Test for Necessary Condition Analysis

Dul

Laan

Kuik

2018

Organizational Research Methods

268

169

View full text Add to dashboard Cite

In this article, we present a statistical significance test for necessary conditions. This is an elaboration of necessary condition analysis (NCA), which is a data analysis approach that estimates the necessity effect size of a condition X for an outcome Y. NCA puts a ceiling on the data, representing the level of X that is necessary (but not sufficient) for a given level of Y. The empty space above the ceiling relative to the total empirical space characterizes the necessity effect size. We propose a statistical significance test that evaluates the evidence against the null hypothesis of an effect being due to chance. Such a randomness test helps protect researchers from making Type 1 errors and drawing false positive conclusions. The test is an "approximate permutation test." The test is available in NCA software for R. We provide suggestions for further statistical development of NCA.Keywords null hypothesis testing, permutation test, necessary condition analysis, statistical significance, p value Necessary condition analysis (NCA; Dul, 2016) is a tool for researchers to develop and test necessary but not sufficient conditions. A necessary condition enables the outcome when present and constrains the outcome when absent. NCA assumes that outcome Y is bound by condition X by drawing a ceiling line on top of the data in an XY scatter plot. The line defines the empty space in the upper left corner of the scatter plot.1 This empty space suggests that high values of Y are not possible with low values of X and indicates that X constrains Y. The size of the empty space relative to the total space with observations reflects the extent of the constraint that X poses on Y: The larger the empty space, the more X constrains Y. The necessity effect size (d) is the size of the empty space above the ceiling as a fraction of the total space where cases are observed or could be observed given by the minimum and maximum empirical or theoretical values of X and Y (scope 2 ). NCA's effect size d has values between 0 and 1.

show abstract

Section: Permutation Testmentioning

confidence: 99%

A Statistical Significance Test for Necessary Condition Analysis

Dul

Laan

Kuik

2018

Organizational Research Methods

268

169

View full text Add to dashboard Cite

show abstract

“…Suppose that the local null hypothesis

H_{0}^{v} : ρ ((),, X, Y_{v}) = 0

is true and that the following three additional assumptions are satisfied: (a) E ( X 2 ) < ∞, (b)

E ((), Y_{v}^{2}) < \infty

, and (c)

E ((), X^{2} Y_{v}^{2}) < \infty

. Given these assumptions, the test statistic in Equation 11 asymptotically follows a normal distribution with mean zero and variance

τ_{v}^{2} = \frac{E ([], {(X - μ_{X})}^{2} {(Y_{v} - μ_{Y_{v}})}^{2})}{σ_{X}^{2} σ_{Y_{v}}^{2}} .

However, the permutation distribution asymptotically follows a standard normal distribution, see DiCiccio and Romano (). If X and Y v are dependent, then

τ_{v}^{2}

could be smaller or larger than one, which implies that the asymptotic standard deviation of the permutation distribution could be too large (

τ_{v}^{2} < 1

) or too small (

τ_{v}^{2} > 1

).…”

Section: Association Testmentioning

confidence: 99%

“…Under the previously stated assumptions (a)–(c), plus the additional two assumptions (d) E ( X 4 ) < ∞ and (e)

E ((), Y_{v}^{4}) < \infty

, the permutation distribution of the studentized test statistic in Equation 12 will asymptotically follow a standard normal distribution when the null hypothesis is true (see DiCiccio & Romano, ). Thus, using the studentized test statistic

T_{v}^{*}

, the independence permutation testing procedure provides an asymptotically valid level α test of the null hypothesis

H_{0}^{v} : ρ ((),, X, Y_{v}) = 0

, which is exact when X and Y v are independent.…”

Section: Association Testmentioning

confidence: 99%

“…Consider a permutation test as described in the previous section, that is, by considering

Y_{v}^{π}

or otherwise X π = [ X π 1 , …, X π p ] with

X_{bold-italicπ j} = {(X_{π_{1} j} \dots X_{π_{n} j})}^{⊤}

for all possible

bold-italicπ \in P_{n}

. Under assumptions A1–A3, DiCiccio and Romano () showed that the permutation distribution of

\sqrt{n} {\overset{false^}{bold-italicβ}}_{v}

is asymptotically normal with mean zero and covariance matrix

Φ_{v}^{⋆} = E ((), {normalɛ}_{italiciv}^{2}) Σ^{- 1}

when the local null hypothesis

H_{0}^{v}

is true. Note that

Φ_{v} = Φ_{v}^{⋆}

if and only if

Ω_{v} = E ((), {normalɛ}_{italiciv}^{2}) boldΣ

, which is true if ɛ iv and X i are independent.…”

Section: Regression Test: All Slopesmentioning

confidence: 99%

“…However, if ɛ iv and X i are dependent, then Φ v will not necessarily be equal to

Φ_{v}^{⋆}

and the permutation test using the F v test statistic in Equation 17 may not be exact or asymptotically valid. Assuming A1–A4, the W v test statistic in Equation 18 should provide an (inexact but) asymptotically level α test of the local null hypothesis when ɛ iv and X i are dependent (see DiCiccio & Romano, ).…”

Section: Regression Test: All Slopesmentioning

confidence: 99%

See 2 more Smart Citations

Statistical nonparametric mapping: Multivariate permutation tests for location, correlation, and regression problems in neuroimaging

Helwig

2019

WIREs Computational Stats

View full text Add to dashboard Cite

Nonparametric statistical inference via permutation testing is on the rise in neuroimaging research. This rise in popularity is likely in response to recent studies that have demonstrated limitations of parametric inference in certain situations. Nonparametric tests have the appeal of requiring fewer assumptions than their parametric counterparts, and are often touted as being more flexible and more useful for small samples. Furthermore, recent studies have demonstrated the robustness of nonparametric methods in situations when parametric inference fails. As a result, many nonstatistical neuroimaging researchers are likely to believe that nonparametric permutation tests are always a "safe choice" because the results do not depend on distributional assumptions and/or large sample approximations. Alas, this commonly held belief is not entirely accurate, given that nonparametric tests still do rely on assumptions and/or approximations for valid statistical inference. When these assumptions are met, nonparametric permutation tests have the potential to produce valid inferential results for the intended hypotheses. However, as I demonstrate, when these assumptions are violated, nonparametric permutation tests can produce invalid and/or misleading results, which have important implications for the use of such methods in neuroimaging research. All hope is not lost though, as recent theoretical developments in nonparametric statistics can improve current implementations of permutation tests in neuroimaging research.

show abstract

Limitation of permutation‐based differential correlation analysis

Song,

2023

Genetic Epidemiology

View full text Add to dashboard Cite

The comparison of biological systems, through the analysis of molecular changes under different conditions, has played a crucial role in the progress of modern biological science. Specifically, differential correlation analysis (DCA) has been employed to determine whether relationships between genomic features differ across conditions or outcomes. Because ascertaining the null distribution of test statistics to capture variations in correlation is challenging, several DCA methods utilize permutation which can loosen parametric (e.g., normality) assumptions. However, permutation is often problematic for DCA due to violating the assumption that samples are exchangeable under the null. Here, we examine the limitations of permutation‐based DCA and investigate instances where the permutation‐based DCA exhibits poor performance. Experimental results show that the permutation‐based DCA often fails to control the type I error under the null hypothesis of equal correlation structures.

show abstract

Robust Permutation Tests For Correlation And Regression Coefficients

Abstract: This document provides additional examples and results for the authors' paper "Robust Permutation Tests For Correlation And Regression Coefficients".

Cited by 77 publications

References 15 publications

A Statistical Significance Test for Necessary Condition Analysis

A Statistical Significance Test for Necessary Condition Analysis

Statistical nonparametric mapping: Multivariate permutation tests for location, correlation, and regression problems in neuroimaging

Limitation of permutation‐based differential correlation analysis

Contact Info

Product

Resources

About