Evaluation of reported claims of sex-based differences in treatment effects across meta-analyses: A meta-research study

Abstract

ImportanceDifferences in treatment effects between men and women may be important across diverse interventions and diseases.ObjectiveWe aimed to evaluate claims of sex-based differences in treatment effects across published meta-analyses.Data SourcesPubMed (searched up to January 17, 2024).Study SelectionPublished meta-analyses of randomized controlled trials (RCTs) that had any mention of sex (male/female) subgroup or related analysis in their abstract.Data Extraction and SynthesisWe determined how many meta-analyses had made claims of sex-based differences in treatment effects. These meta-analyses were examined in depth to determine whether the claims reflected sex-treatment interactions with statistical support or fallacious claims and categorized the frequency of different fallacies. For claims with statistical support, we examined whether they were considered and discussed in UpToDate. Whenever possible, we re-analyzed the p-value for sex-treatment interaction.Main Outcomes and MeasuresNumber of claims with statistical support and fallacious claims; clinical implications of subgroup differences.Results216 meta-analysis articles fulfilled the eligibility criteria. Of them, 99 stated in the abstract that that there was no sex-based difference, and 20 mentioned a sex-based subgroup analysis without reporting results in the abstract. 97 meta-analyses made 115 claims of sex-based differences. Of them, 27 claims across 21 articles had statistical support at p<0.05. 4/27 claims were mentioned in UpToDate, but none led to different recommendations for men and women. 35 articles had 39 fallacious claims where the sex-treatment interaction was not statistically significant (significant effects in one sex (29 claims in 25 articles), larger effects in one sex (7 claims in 7 articles), other (3 claims in 3 articles)). Another 44 articles made claims based on potentially fallacious methods (39 based on meta-regression of percentage of one group and 5 providing the results of only one group), but proper data were unavailable to assess statistical significance.Conclusions and relevanceFew meta-analyses of RCTs make claims of sex-based differences in treatment effects and most of these claims lack formal statistical support. Statistically significant and clinically actionable sex-treatment interactions may be rare.