Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery

Abstract

Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgeryBackground: Randomized controlled trials (RCTs) are thought to provide the most accurate estimation of "true" treatment effect. The relative quality of effect estimates derived from nonrandomized studies (nRCTs) remains unclear, particularly in surgery, where the obstacles to performing high-quality RCTs are compounded. We performed a meta-analysis of effect estimates of RCTs comparing surgical procedures for breast cancer relative to those of corresponding nRCTs.Methods: English-language RCTs of breast cancer treatment in human patients published from 2003 to 2008 were identified in MEDLINE, EMBASE and Cochrane databases. We identified nRCTs using the National Library of Medicine's "related articles" function and reference lists. Two reviewers conducted all steps of study selection. We included studies comparing 2 surgical arms for the treatment of breast cancer. Information on treatment efficacy estimates, expressed as relative risk (RR) for outcomes of interest in both the RCTs and nRCTs was extracted. Results:We identified 12 RCTs representing 10 topic/outcome combinations with comparable nRCTs. On visual inspection, 4 of 10 outcomes showed substantial differences in summary RR. The pooled RR estimates for RCTs versus nRCTs differed more than 2-fold in 2 of 10 outcomes and failed to demonstrate consistency of statistical differences in 3 of 10 cases. A statistically significant difference, as assessed by the z score, was not detected for any of the outcomes. Conclusion:Randomized controlled trials comparing surgical procedures for breast cancer may demonstrate clinically relevant differences in effect estimates in 20%-40% of cases relative to those generated by nRCTs, depending on which metric is used. Contexte :On estime que les essais randomisés et contrôlés (ERC) offrent l'estimation la plus juste de l'effet « réel » des traitements. La qualité relative de l'estimation de l'effet dérivée des études non randomisées (EnRC) reste indéterminée, particulièrement en ce qui concerne la chirurgie, où les obstacles à la réalisation d'ERC optimaux sont nombreux. Nous avons procédé à une méta-analyse des estimations de l'effet dérivées d'ERC qui comparaient des interventions chirurgicales pour le cancer du sein, aux estimations dérivées d'EnRC assortis. Méthodes :Nous avons relevé les ERC sur le traitement du cancer du sein chez l'humain publiés en langue anglaise entre 2003 et 2008 à partir des bases de données MEDLINE, EMBASE et Cochrane. Nous avons relevé les EnRC à l'aide des listes bibliographiques et de la fonction « articles connexes » de la National Library of Medicine. Deux examinateurs se sont chargés de toutes les étapes de la sélection des études. Nous avons inclus les études qui comparaient 2 groupes soumis à une chirurgie pour le traitement du cancer du sein. Nous avons extrait l'information sur les estimations de l'efficacité des traitements, exprimées sous forme de risque relatif (RR) de survenue des paramètres r...

Study methodology in trauma care: towards question-based study designs

“…Shikata et al performed a similar comparison between research design, but restricted the comparison to 18 digestive surgery topics and found a significant difference between different study designs (observational vs RCT) in 25% of the primary outcomes assessed [ 20 ]. Similar findings were presented more recently by Edwards et al for breast cancer surgery; they found differences in study results between designs in 2 out of 10 topics [ 21 ]. The overall conclusion, based on theoretical rather than empirical considerations, seems to be that RCTs have superior validity (notably due to the randomization), yet in certain situations the different designs may yield actually quite similar results.…”

Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples

“…Results from randomized trials (9,(13)(14)(15) and observational studies (6,9,10,(13)(14)(15)(16)(17)(18)(19)(20)(21) led to the now widely accepted conclusion that ART should be initiated as soon as possible after diagnosis of HIV infection. The two most recent studies, the randomized START trial (9) and the observational HIV-CAUSAL Collaboration(10), compared the effectiveness of immediate initiation regardless of CD4 count versus deferred initiation until CD4 count dropped below 350 cells/mm 3 or acquired immunodeficiency syndrome (AIDS) was diagnosed in HIV-positive, AIDS-free, and treatment-naïve individuals with CD4 count>500 cells/mm 3 at the start of the study.…”

“…If differences other than randomization are not explicitly taken into account, randomized-observational comparisons, as commonly undertaken in meta-analyses (4)(5)(6)(7)(8), may be hard to interpret because they generally compare "apples with oranges" rather than "apples with apples". Informative comparisons between randomized and observational estimates will often require a careful re-analysis of the data of both the randomized trials and the observational studies.…”

Clinical Research in Surgery: Threats and Opportunities

“…Other notable examples include the Halsted radical mastectomy, kidney decapsulation to treat hypertension, and uterine suspension. A recent comparison of randomized and nonrandomized studies in breast cancer surgery found that, depending on the metric used, in 20-40% of outcomes the effect estimates differed more than twice between both study types, calling into question the validity of evidence derived from nonrandomized comparisons [19]. Similar findings were reported by Peinemann et al [20], who performed a systematic review of methodological studies examining the effect of the study type on the reported results.…”