Reporting of numerical and statistical differences in abstracts

Abstract

OBJECTIVE:The reporting of relative risk reductions (RRRs) or absolute risk reductions (ARRs) to quantify binary outcomes in trials engenders differing perceptions of therapeutic efficacy, and the merits of P values versus confidence intervals (CIs) are also controversial. We describe the manner in which numerical and statistical difference in treatment outcomes is presented in published abstracts. DESIGN:A descriptive study of abstracts published in 1986 and 1996 in 8 general medical and specialty journals. Inclusion criteria: controlled, intervention trials with a binary primary or secondary outcome. Seven items were recorded: raw data (outcomes for each treatment arm), measure of relative difference (e.g., RRR), ARR, number needed to treat, P value, CI, and verbal statement of statistical significance. The prevalence of these items was compared between journals and across time. RESULTS:Of 5,293 abstracts, 300 met the inclusion criteria. In 1986, 60% of abstracts did not provide both the raw data and a corresponding P value or CI, while 28% failed to do so in 1996 (P < .001; RRR of 53%; ARR of 32%; CI for ARR 21% to 43%). The variability between journals was highly significant (P < .001). In 1986, 100% of abstracts lacked a measure of absolute difference while 88% of 1996 abstracts did so (P < .001). In 1986, 98% of abstracts lacked a CI while 65% of 1996 abstracts did so (P < .001). CONCLUSIONS:The provision of quantitative outcome and statistical quantitative information has significantly increased between 1986 and 1996. However, further progress can be made to make abstracts more informative.KEY WORDS: periodicals; abstracts; statistics; format effect; quantitative results. J GEN INTERN MED 2002;17:203±206. T he abstract has increasingly become a crucial source of information for the busy physician accessing the medical literature. 1 It is often the means by which articles to be read are selected and in cases in which the full text is unavailable, it is even the basis upon which clinical decisions are made. 2 Concern about the need for more adequate abstracts of research papers led an ad hoc working group of clinical epidemiologists and journal editors to publish guidelines 3 for structured abstracts in April, 1987. These guidelines have since been modified and refined, 4 and a particular emphasis has been placed on the presentation of study results. 5,6 Despite these efforts to improve the informative content of abstracts, the data presented in them may still lack quantitative measures of statistical significance (P value and confidence interval [CI]) and may be misinterpreted by readers due to the effect of the outcome format on the appraisal of trial results. 7 Format effects are particularly salient in the setting of a low-event rate (e.g., 2% death rate), where a large relative risk reduction (RRR) (``50% fewer deaths'') corresponds to only a small absolute risk reduction (ARR)(``1% fewer persons die'') and a large number needed to treat (NNT) (``need to treat 100 people to save 1 life''). A high RR...

A Meta-analysis of the Effects of Presenting Treatment Benefits in Different Formats

“…This message is important not only in terms of improving our understanding of the methodological reasons for variability in the framing effects observed between studies but also for physician-patient communication. 13 As well as raising awareness of the potential deficits in the way that clinical trial data presented in the medical literature, drug advertisements, and news media, [14][15][16][17][18][19] this review has highlighted the dangers of providing patients with relative risk reduction information in isolation. It might therefore be argued that to avoid the possibility of misleading patients about the benefits of a treatment, physicians should put the size of the relative changes into perspective by presenting either baseline risk…”

Endpoint Selection and Relative (Versus Absolute) Risk Reporting in Published Medication Trials

“…6 Another notable finding of our study is that authors commonly report trial results exclusively in relative terms despite a growing recognition that relative risk reporting can distort the magnitude of the effects of an intervention. 13,14,18,19 Given these concerns, readers are likely to be better served by uniform inclusion of absolute risk reporting.…”

“…In addition, the study did not report whether trials using composite endpoints were more or less likely than trials not using composite endpoints to report significant results. Therefore, it is not clear that these results contradict our finding that trials using composite endpoints are less likely than trials not using composite endpoints to show positive results Disease-Specific Mortality Kip et al 17 An analysis of 20 trials comparing bare metal vs. drug-eluting stents showed that 55% used cardiovascular mortality as an endpoint while 45% used all-cause mortality; another analysis reported in the same paper showed that, among 27 trials published in 2006 in the Journal of the American College of Cardiology, 19% used cardiovascular death as an endpoint while 74% used all-cause mortality Relative Risk Reporting Nuovo et al 14 Fewer than 10% of trials published in top medical journals prior to 1998 reported risk reductions in absolute terms anywhere in the article Dryver et al 18 Among articles published in eight journals in 1996, 12% of the abstracts provided a measure of absolute risk difference; no abstracts published in these journals in 1986 did Schwartz et al 19 Among articles published in 2003 and 2004 in six top journals, 68% of failed to report the underlying absolute risks for the first ratio measure in the abstract and especially since this finding was of borderline significance, the association may represent a chance occurrence. We did not detect other associations between trial funding and endpoint selection or relative risk reporting or between endpoint selection and trial outcomes, however our power to detect some of these associations was limited.…”

“…In particular, experts have suggested that the use of surrogate endpoints, [2][3][4][5][6] and composite endpoints, [7][8][9][10][11] and the use of disease-specific-rather than all-cause-mortality as an endpoint, 12,13 may mislead readers if the limitations of these endpoints are not adequately explained. Similar concerns have also been raised about the reporting of trial results in relative-rather than absolute-numbers, 14,15 Table 1 describes these concerns. A limited number of studies have examined the prevalence of the use of surrogate [3][4][5] and composite endpoints 7,9,16 the use of disease specific mortality as an endpoint 12,17 and the reporting of results in relative numbers 14,18,19 in clinical trials. However, data from these studies are several years old, typically reflect trials involving treatments for only a limited spectrum of diseases, and provide only limited details about factors that may be associated with these endpoints and with relative risk reporting.…”

Evidence-based research: understanding the best estimate

“…From these data, relative effect measures may be chosen and calculated by readers, researchers, or both and reported along with the absolute measures. 4 …”