Significance Test, Confidence Interval, Both or Neither?

Abstract

It is reasonably well-known that you can get a statistical significance test "for free" by constructing a confidence interval around an obtained statistic and seeing whether or not the corresponding hypothesized parameter is "captured" by the interval. If it is not inside the 95% confidence interval, for example, reject it at the .05 significance level and conclude that the sample finding is statistically significant. If it is, do not reject it; the sample finding is not statistically significant at that level. Therefore, if you want a significance test you can either carry it out directly or get it indirectly via the corresponding confidence interval. If you want a confidence interval you can carry it out directly (the usual way) or you can get it indirectly by carrying out significance tests for all of the possible "candidates" for the hypothesized parameter (not very practical, as there is an infinite number of them). But should you ever carry out a hybrid combination of hypothesis testing and interval estimation, for example, by reporting the 95% confidence interval and also reporting the actual p value that "goes with" the obtained statistic, even if it is greater than or less than .05? Some researchers do that. Some journals (e.g., The Journal of Managed Care and Specialty Pharmacy) require it. At least one journal (Basic and Applied Social Psychology) has banned both. It is also reasonably well-known that if you do not have a random sample, you really should not make any statistical inferences. Instead, get the descriptive statistic(s) and make any nonstatistical inferences warranted. Exception: If you have random assignment but not random sampling for an experiment, randomization tests (permutation tests) are fine, but the inference is to all

On the Performance of Bayesian Generalized Dissimilarity Model Estimator

“…In Table 1, the interval estimations produced by these models indicate the uncertainty predictions regarding biological dissimilarity in the Dune dataset. Moreover, they can also be used for significance testing, where an interval estimate for a model parameter not containing zero suggests the significance of that parameter, i.e., the significant effect of the predictor variable corresponding to the model parameter on the response variable [24]. Therefore, based on Table 1, it can be said that in the BBGDM, soil thickness does not significantly affect biological dissimilarity.…”

Use of 95% confidence intervals in the reporting of between-group differences in randomized controlled trials: analysis of a representative sample of 200 physical therapy trials

“…Even when p-values are interpreted correctly, they do not portray some crucial information about the magnitude or the clinical relevance of the difference between the groups. 5,6 Thus, a statistically significant finding should not be interpreted on its own to influence clinical practice. 6,7 The p-value also provides no information about the uncertainty around the trial's estimate of the effect of the intervention.…”