Are All Significant P Values Created Equal?

Abstract

Just as diagnostic tests are most helpful in light of the clinical presentation, statistical tests are most useful in the context of scientific knowledge. Knowing the specificity and sensitivity of a diagnostic test is necessary, but insufficient: the clinician must also estimate the prior probability of the disease. In the same way, knowing the P value and power, or the confidence interval, for the results of a research study is necessary but insufficient: the reader must estimate the prior probability that the research hypothesis is true. Just as a positive diagnostic test does not mean that a patient has the disease, especially if the clinical picture suggests otherwise, a significant P value does not mean that a research hypothesis is correct, especially if it is inconsistent with current knowledge. Powerful studies are like sensitive tests in that they can be especially useful when the results are negative. Very low P values are like very specific tests; both result in few false-positive results due to chance. This Bayesian approach can clarify much of the confusion surrounding the use and interpretation of statistical tests.

Economic evaluation of ibuprofen/codeine combination versus ibuprofen alone for acute pain treatment in Peru

“…Although the study involved only 25 patients, arguably too few for the study to have much predictive power, [7] the cross‐over design in analgesic studies increases sensitivity by 40% compared with parallel group designs, [8] so that this cross‐over would have sensitivity equivalent to a parallel group study with 40 patients in each group.…”

Historical Origins of Statistical Testing Practices

“…Since the 1960s, the roles and pra~tices of statistical testing in quantitative research have been discussed in a nu~ber of substantive disciplines; for example, in economics (Good, 1981;Mayo, 1~81), education (Carver, 1978;Orey, Garrison, & Burton, 1989), marketing (S~wyer & Peter, 1983), medicine (Browner & Newman, 1987), occupational thera~y (Ottenbacher, 1985), psychology (Meehl, 1978;Rosnow & Rosenthal, 1989;Se~lin, 1987), social research (Guttman, 1985), and sociology (Gold, 1969). Testing is also currently a topic of interest to methodologists; recent articles have appeared dealing with new ideas (e.g., Simon & Bruce, 1991, on using bootstrap m~thods to validate test results), with the rehashing of old ideas (e.g., Pillemer, •1991), and with the teaching of statistical testing (e.g., Bangdiwala, 1989).…”

Fifty years of progress in cardiology: a personal perspective.

“…Even though many such trials have provided valuable data and therapeutic guidance, several papers have appeared pointing out the difficulty of designing trials with genuine relevance to clinical practice, advocating caution in interpretion of their results. [71][72][73][74][75] Modern cardiology faces many problems. Technologic advances are associated with great expense, which is compounded by rapid obsolescence of equipment already purchased.…”