The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews

Abstract

Meta-analyses summarize a field’s research base and are therefore highly influential. Despite their value, the standards for an excellent meta-analysis, one that is potentially award-winning, have changed in the last decade. Each step of a meta-analysis is now more formalized, from the identification of relevant articles to coding, moderator analysis, and reporting of results. What was exemplary a decade ago can be somewhat dated today. Using the award-winning meta-analysis by Stahl et al. (Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies, 41(4):690–709, 2010) as an exemplar, we adopted a multi-disciplinary approach (e.g., management, psychology, health sciences) to summarize the anatomy (i.e., fundamental components) of a modern meta-analysis, focusing on: (1) data collection (i.e., literature search and screening, coding), (2) data preparation (i.e., treatment of multiple effect sizes, outlier identification and management, publication bias), (3) data analysis (i.e., average effect sizes, heterogeneity of effect sizes, moderator search), and (4) reporting (i.e., transparency and reproducibility, future research directions). In addition, we provide guidelines and a decision-making tree for when even foundational and highly cited meta-analyses should be updated. Based on the latest evidence, we summarize what journal editors and reviewers should expect, authors should provide, and readers (i.e., other researchers, practitioners, and policymakers) should consider about meta-analytic reviews.

Is Internationalization Beneficial to Innovation? Evidence from a Meta-analysis

“…This study examines the moderating effect of these key contextual factors and thereby provides evidence on their impact on the potency of estimates. Evidence on the importance of these contextual factors helps to identify some of the major elements that determine the boundary conditions in which the postulated relationships from causal paths apply (Steel et al, 2021). Evidence that key contextual factors have meaningful impact on estimates provide grounds for a more comprehensive addressing of these factors in the conceptualizations of causal chains in the various theoretical approaches used to investigate the nexus.…”

“…Unfortunately, there are few studies published in high quality journals that provide such information. This currently makes it difficult to assess the robustness of estimates from different theoretical approaches and from different causal chains using the same theoretical approach (Steel et al, 2021). Future research therefore needs to provide clear indications of the theoretical approach, causal relationships that are used, the contextual factors that are considered and how they relate to the postulated causal chains.…”

“…Our study responds to calls to improve research design to enhance the validity and reliability of results (Dimos & Pugh, 2016;Starbuck, 2016). These calls are also growing in International Business research (e.g., Aguinis et al, 2017;Cuervo-Cazurra et al, 2016;Reeb et al, 2012;Steel et al, 2021). Good research design includes addressing important contextual factors and analytical techniques to improve the validity and robustness of results.…”

International business studies: Are we really so uniquely complex?

“…For future directions, the authors called for research using more dynamic process-oriented moderator variables, such as boundary-spanning activities. This article was also particularly exemplary for its time in the use of meta-analysis, a method used not only in IB but also extensively in OB and other fields (Steel et al, 2021). Meta-analysis allowed for detecting moderating effects not testable in the primary studies, which is especially important for reconciling past results relying on different operationalizations of variables and study designs.…”

Relational Capabilities under Moderating and Mediating Effects to Understand their Impacts on Firm Financial Performance

“…Following recommendations for meta-analysis steps (Steel, Beugelsdijk, & Aguinis, 2021) a code protocol was developed to capture the measures of variables and reduce errors in the coding process in order to ensure consistency in the extraction of information. The measures of variables used were identified and analyzed based on their similarities with the code protocol Table 1.…”