What Are the Consequences If the Assumption of Independent Observations Is Violated in Reliability Generalization Meta-Analysis Studies?

Romano, Jeanine; Kromrey, Jeffrey D.

doi:10.1177/0013164408323237

Cited by 15 publications

(16 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scholars therefore suggest that minimizing the lack of independence might be the best that can be expected (Cooper & Koenka, ; Schmidt & Oh, ). Various simulation analyses support this approach and find negligible effects of statistical independence violations (Bijmolt & Pieters, ; Romano & Kromrey, ; Tracz, Elmore, & Pohlmann, ).…”

Section: Methods and Datamentioning

confidence: 95%

The Robustness of the Corporate Social and Financial Performance Relation: A Second‐Order Meta‐Analysis

Busch

Friede

2018

Corp Soc Responsibility Env

246

147

View full text Add to dashboard Cite

For many decades, there has been a debate about the relation between corporate social/environmental performance (CSP) and corporate financial performance (CFP). Our study presents a review of academic research on this topic by applying a second-order meta-analysis. The data sample combines 25 previous meta-analyses yielding a sample size of one million observations. Our results demonstrate a highly significant, positive, robust, and bilateral CSP-CFP relation. The relation is positive regardless of whether firms focus on ecological or social aspects, though corporate reputation turns out to be a key CSP determinant. We find a particularly strong CSP-CFP relation for operational CFP. Furthermore, we add a new perspective on potential biases resulting from the studies' publishing source: social issues-oriented journals and methodological weaker papers do not distort the positive CSP-CFP relation. Our conclusion is: Based on the extant literature, the business case for being a good firm is undeniable. found a significant, positive, and bidirectional correlation of CSP with CFP. Despite these findings, the authors claim that this relation cannot be generalized across all CSP dimensions and CFP categories, detecting a large variabilityeven after correcting for sampling error and measurement error. Since then, the picture has become even more ambiguous and critical debates have resurfaced about the actual CSP-CFP relation (one hand, many scholars claim that, to date, the literature on the CSP-CFP relation remains inconclusive. On the other hand, a whole range of new studies and further meta-analyses have been published supporting a positive relation. Thus, more than a decade after Orlitzky et al. (2003), it seems to be the right time to revive the paper's spirit and ask: when generalizing existing research, is there still evidence that it pays off to be a good firm?To answer this question, we apply a second-order meta-analysis. In other scientific disciplines like medicine, psychology, or education (i.e. research areas typically also experiencing vast amounts of data and conflicting results), meta-analytic techniques are widespread and have been applied since the 1960s (Glass, 1976;Hedges & Olkin, 1985;Hunter, Schmidt, & Jackson, 1982). With the growing number of meta-analyses, the idea of cumulative meta-analysis also appeared (Kazrin, Durac, & Agteros, 1979). Several labels have been used for such analyses: overview of reviews, review of reviews, umbrella reviews, meta-reviews, cumulative meta-analyses, meta-meta-analysis, and second-order meta-analysis (Cooper & Koenka, 2012). However, until recently, no standard procedure with comparable rigour to first-order meta-analytical approaches had emerged. This paper makes use of the novel method for conducting second-order meta-analysis proposed by Schmidt and Oh (2013).We contribute to the academic inquiry on the CSP-CFP relation in several regards. By summarizing the findings of existing meta-analyses and utilizing a data sample that is more than 25 times larger compare...

show abstract

Section: Methods and Datamentioning

confidence: 95%

The Robustness of the Corporate Social and Financial Performance Relation: A Second‐Order Meta‐Analysis

Busch

Friede

2018

Corp Soc Responsibility Env

246

147

View full text Add to dashboard Cite

show abstract

“…W. L. Cheung, 2014; S. F. Cheung & Chan, 2004; Cooper, 1979; Greenhouse & Iyengar, 1994; Hedges & Olkin, 1985; Landman & Dawes, 1982; Marín-Martínez & Sánchez-Meca, 1999; Raudenbush et al, 1988; Romano & Kromrey, 2009; Rosenthal & Rubin, 1986; Tracz et al, 1992). To help researchers address this violation of independence assumption in their meta-analyses, we presented an integration of multilevel and meta-analytic models as well as provided illustrative examples highlighting how modeling dependence in primary effect sizes results in differing effect size estimates and confidence bands.…”

Section: Discussionmentioning

confidence: 99%

“…That is, all these techniques assume that the primary effect sizes are independent of one another. This is quite a serious issue, and Romano and Kromrey (2009) summarized it as follows:The assumption regarding independence of observations is commonly violated in meta-analytic research (Becker, 2000; Hedges & Olkin, 1985; Hunter & Schmidt, 1990)…. Furthermore, several studies have been conducted concerning the consequences of dependent observations in meta-analysis (e.g., Becker & Kim, 2002; Beretvas & Pastor, 2003; Cooper, 1979; Greenhouse & Iyengar, 1994; Hedges & Olkin, 1985; Landman & Dawes, 1982; Raudenbush, Becker, & Kalaian, 1988; Rosenthal & Rubin, 1986; Tracz, Elmore, & Pohlmann, 1992).…”

mentioning

confidence: 99%

Meta-Analyses as a Multi-Level Model

Gooty

Banks

Loignon

et al. 2019

Organizational Research Methods

View full text Add to dashboard Cite

Meta-analyses are well known and widely implemented in almost every domain of research in management as well as the social, medical, and behavioral sciences. While this technique is useful for determining validity coefficients (i.e., effect sizes), meta-analyses are predicated on the assumption of independence of primary effect sizes, which might be routinely violated in the organizational sciences. Here, we discuss the implications of violating the independence assumption and demonstrate how meta-analysis could be cast as a multilevel, variance known (Vknown) model to account for such dependency in primary studies’ effect sizes. We illustrate such techniques for meta-analytic data via the HLM 7.0 software as it remains the most widely used multilevel analyses software in management. In so doing, we draw on examples in educational psychology (where such techniques were first developed), organizational sciences, and a Monte Carlo simulation (Appendix). We conclude with a discussion of implications, caveats, and future extensions. Our Appendix details features of a newly developed application that is free (based on R), user-friendly, and provides an alternative to the HLM program.

show abstract

“…Due to the addition of a second variance component, the random-effects model provides more conservative results than the fixed-effect model (Beretvas & Pastor, 2003). Nonetheless, several simulation studies have warned about the limitations of the fixed-effect model in general meta-analysis (e.g., Brockwell & Gordon, 2001; Marín-Martínez & Sánchez-Meca, 2010) and in the RG approach (Bonett, 2010; Romano & Kromrey, 2009).…”

mentioning

confidence: 99%

Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach

López-López

Botella

Meca

et al. 2013

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

Since heterogeneity between reliability coefficients is usually found in reliability generalization studies, moderator analyses constitute a crucial step for that meta-analytic approach. In this study, different procedures for conducting mixed-effects meta-regression analyses were compared. Specifically, four transformation methods for the reliability coefficients, two estimators of the residual between-studies variance, and two methods for testing regression coefficients significance were combined in a Monte Carlo simulation study. The different methods were compared in terms of bias and mean square error (MSE) of the slope estimates, and Type I error and statistical power rates for the slope statistical tests. The results of the simulation study did not vary as a function of the residual variance estimator. All transformation methods provided negatively biased estimates, but both bias and MSE were reasonably small in all cases. In contrast, important differences were found regarding statistical tests, with the method proposed by Knapp and Hartung showing a better adjustment to the nominal significance level and higher power rates than the standard method.

show abstract

What Are the Consequences If the Assumption of Independent Observations Is Violated in Reliability Generalization Meta-Analysis Studies?

Cited by 15 publications

References 42 publications

The Robustness of the Corporate Social and Financial Performance Relation: A Second‐Order Meta‐Analysis

The Robustness of the Corporate Social and Financial Performance Relation: A Second‐Order Meta‐Analysis

Meta-Analyses as a Multi-Level Model

Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach

Contact Info

Product

Resources

About