Stepwise Multiple Testing as Formalized Data Snooping

Abstract

In econometric applications, often several hypothesis tests are carried out at once. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. This paper suggests a stepwise multiple testing procedure that asymptotically controls the familywise error rate. Compared to related single-step methods, the procedure is more powerful and often will reject more false hypotheses. In addition, we advocate the use of studentization when feasible. Unlike some stepwise methods, the method implicitly captures the joint dependence structure of the test statistics, which results in increased ability to detect false hypotheses. The methodology is presented in the context of comparing several strategies to a common benchmark. However, our ideas can easily be extended to other contexts where multiple tests occur. Some simulation studies show the improvements of our methods over previous proposals. We also provide an application to a set of real data. Copyright The Econometric Society 2005.

A non-parametric test for industrial specialization

“…Though well established in statistics and biostatistics, economists have only recently began to recognize and properly correct for the flawed inference due to Type I errors under multiple hypothesis testing in empirical research. In recent work, Romano and Wolf (2005) stressed the need to minimize empirical data snooping for 'false positives' by controlling for familywise error rates (p F W E ). This entails adjusting the critical values for each of the individual hypothesis tests to ensure that the probability of rejecting the null for any one of the multiple hypothesis tests is approximately equal to the p F W E .…”

Controlling the Danger of False Discoveries in Estimating Multiple Treatment Effects

“…That is, individual testing labels three treatment effects out of six as statistically significant. By putting a 5% threshold on the risk of making one or more false discoveries, these three individually significant treatment effects are indicated as false discoveries by multiple testing methods from Romano and Wolf (2005).…”

“…The multiple testing method of Romano and Wolf (2005) based on bootstrapping essentially solves the following problem MTest: Find one critical value c M T est 1−α such that familywise error rate FWE ≤ α asymptotically.…”

“…Control of the FDP is achieved by increasing k within 'probability of falsely rejecting k or more null hypotheses' until a criterion is met; see Romano and Wolf (2005) for details.…”

“…I reexamine the empirical model of Lalive et al (2005) with respect to making false discoveries. I control the risk of labeling nonexistent treatment effects as (individually) significant by using the powerful multiple testing methods from Romano and Wolf (2005).…”