Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria.

Abstract

This article identifies some potential problems in meta-analysis research and suggests 14 specific criteria for evaluating the quality of meta-analysis research. The analysis is based on a review and replication of a meta-analysis research study. Our replication of the Terpstra (1981) study on methodological rigor in OD research found no evidence of positive-findings bias in OD. Our review of that research study identified a series of methodological problems that are applicable to other meta-analysis studies. These general problems are discussed, and some solutions are offered.

The value of formal planning for strategic decisions: Reply

“…Recent research has cast doubt on the use of Terpstra's scale. Bullock and Svyantek (1985) attempted to replicate Terpstra's study. Unable to obtain information from Terpstra, they applied his process to the literature that he claimed to have used.…”

Understanding Mail Survey Response Behavior: A Meta-Analysis

“…Despite some limitations, the current research incorporated several aspects of a properly conducted meta-analysis as identified by Bangert-Drowns (1986) and Bullock and Svyantek (1985). Moreover, meta-analysis procedures of Glass (1977), Hunter and Schmidt (1990), and Hunter, Schmidt, and Jackson (1982) were combined in this study.…”

A meta-analysis of the contribution of eye movements in processing emotional memories

“…In addition a single rater selected the studies and calculated the effect sizes which increases bias, particularly given unpublished studies were included. Over the years there has been agreement that meta-analysis should involve multiple raters (Bullock & Sysvyantek, 1985;Stroup et al, 2000). The issue of possible publication bias in the current analysis was examined with funnel plot and Duval and Tweedie"s trim and fill procedure.…”