Six Meta-analyses on Treatments for Femoroacetabular Impingement Syndrome in a Year and Readers Are None the Wiser: Methods Advice for Researchers Planning Meta-analysis of Data From Fewer Than 5 Trials

Abstract

Synopsis In 2020, 6 meta-analyses comparing arthroscopic hip surgery to physical therapy were published. All included the same 3 randomized controlled trials, and none used methods suitable for the analysis of fewer than 5 studies. When there are fewer than 5 studies and heterogeneity, a random-effects model with the Hartung-Knapp-Sidik-Jonkman adjustment for a maximally conservative estimate should be employed; if reliable prior information is available, a Bayesian random-effects meta-analysis should be employed. Our re-analysis, which employed the appropriate model, found that there is currently insufficient evidence to conclude that surgery is superior to physical therapy for femoroacetabular impingement (FAI) syndrome. Further randomized controlled trials are required to resolve the clinical question of what the best treatment approach is for FAI syndrome. We provide readers with tools to conduct appropriate meta-analysis of fewer than 5 trials. J Orthop Sports Phys Ther 2021;51(5):201–203. doi:10.2519/jospt.2021.0107

Primary surgery versus primary rehabilitation for treating anterior cruciate ligament injuries: a living systematic review and meta-analysis

“…Bayesian meta-analysis can be more efficient than frequentist methods if the number of studies is small (≤5 studies) and heterogeneity is present. [40][41][42][43] This is the case if empirical prior distributions for variance of the true effects (τ²) are available, as this allows a better estimation of τ² when few studies are available. 43 Prespecified prior distributions are described in online supplemental appendix 5.…”

Body composition changes following different types of bariatric surgery: Interpreting discordant findings from different methodological approaches

“…4 At least 16 different estimators of between-trial variance have been proposed in the literature, all of which have strengths and weaknesses in different circumstances. [4][5][6] There is currently no consensus regarding the analysis and interpretation of data in meta-analyses of a small number of studies, but the Hartung-Knapp-Sidik-Jonkman (HKSJ) method has been recommended when there are less than five studies. [4][5][6] This recommendation is attributed to the HKSJ method usually providing a better representation of estimate uncertainty/imprecision compared with the DL method, which decreases the risk of making type 1 errors.…”

“…[4][5][6] There is currently no consensus regarding the analysis and interpretation of data in meta-analyses of a small number of studies, but the Hartung-Knapp-Sidik-Jonkman (HKSJ) method has been recommended when there are less than five studies. [4][5][6] This recommendation is attributed to the HKSJ method usually providing a better representation of estimate uncertainty/imprecision compared with the DL method, which decreases the risk of making type 1 errors. 6,7 However, the HKSJ method may also demonstrate limitations when applied in meta-analyses with only two studies.…”

“…[4][5][6] This recommendation is attributed to the HKSJ method usually providing a better representation of estimate uncertainty/imprecision compared with the DL method, which decreases the risk of making type 1 errors. 6,7 However, the HKSJ method may also demonstrate limitations when applied in meta-analyses with only two studies. Gonnermann et al 8 showed in a simulation study that when I 2 = 15% and k = 2, the power of a meta-analysis can be below 15% using the HKSJ method versus approximately 60% using the DL method, with both methods having similar type 1 error rates (HKSJ: 5% vs. DL: 4%).…”

“…Thus, we chose to follow the less conservative approach in our analyses, but acknowledge that future meta-analyses on this topic could present both DL and HKSJ-adjusted summary estimates for body composition changes in meta-analyses of five or fewer studies, and justify reasoning for interpreting one result over the other when HKSJ and DL methods provide discordant results. 6,12 Finally, we would like to thank Cai et al for drawing our attention to the discrepancy between the first and second versions of the Cochrane Risk of Bias (ROB) tool. We acknowledge that we did not complete outcome-based assessments and instead completed study-based assessments.…”