Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments.

Abstract

A series of studies designed to investigate three major aspects of the peer-evaluation system in psychology is presented. Editors and editorial consultants for nine major psychology journals were surveyed for opinions about the desirability of article characteristics. Dimensional structures for evaluation were explored, resulting in a set of prescriptive norms for assessment. Substantial agreement on the desirability of article characteristics is demonstrated, and psychologists heavily involved in the manuscript decision-making processes associated with different journals apparently employ these dimensions in the same way. These results "were used in a second study demonstrating increased reliability of peer judgments of article quality. Finally, it was found that peer judgments of article quality and impact are only very modestly correlated with subsequent citation of the articles,The scientific enterprise in psychology, as in other disciplines, functions as a social system. Determinations of'success, failure, quality, relevance, and prestige are based on a system of peer evaluation. Garvey and Gottfredson (1976) have discussed the importance of peer evaluation and communication to the maintenance of our social system of scientific activity, and Kuhn (1962) offers a theoretical framework within which this process operates. From this perspective, approximations to "truth" are replaced by approximations to the perceived nature of the scientific paradigm; and the author's conviction that his or her view approximates "truth" is of less importance in its evaluation than the peer judgment of its approximation to the paradigm.

Scientific Utopia

“…This put it in the top 25% of general biological science journals and nearly as high as Psychological Science (4.7). This casts further doubt on reviewers' ability to predict importance (Gottfredson, 1978) or at least one indicator of importance: citation impact. With a publishing model focused on soundness, negative results and replications are more publishable, and the journal identity is not defined as publishing research that is otherwise unpublishable.…”

“…Many manuscripts are rejected on this criterion, even if the reviewers identify the research as sound and reported effectively. Despite evidence of the unreliability of the review process for evaluation and identifying importance (Bornmann, Mutz, & Daniel, 2010;Cicchetti, 1991;Gottfredson, 1978;Marsh & Ball, 1989;Marsh, Jayasinghe, & Bond, 2008;Peters & Ceci, 1982;Petty, Fleming, & Fabrigar, 1999;Whitehurst, 1984), this is a reasonable criterion given that journals have limited space and desires to be prestigious outlets. However, in the digital age, page limits are an anachronism (Nosek & Bar-Anan, 2012).…”

How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication

“…However, editors and referees may agree about properties that constitute true value but be unable to perceive true value accurately. Gottfredson (1978), Gottfredson and Gottfredson (1982), and Wolff (1970) found that reviewers for psychological journals agree rather strongly about properties that manuscripts ought to possess, but their judgments about whether specific manuscripts possess various properties agree much less strongly; correlations ranged from 0.16 to 0.50.…”

“…There is no way to directly measure reviewers' abilities to discern manuscripts' true values, but two kinds of observations offer tangential evidence about these abilities. First, Gottfredson (1977Gottfredson ( , 1978 found that reviewers' forecasts of manuscripts' impacts correlated only 0.37 with later citations and their ratings of manuscript quality correlated only 0.24 with later citations. If true values are less visible and more difficult to discern than citation values, correlations of reviewers' judgments with manuscripts' true values are lower than these correlations.…”

“…One of these, discussed near the end of this article, is citation value-the value that researchers see when they choose published works to cite. Citation value is partly a self-fulfilling prophecy because articles in higher-status journals attract more attention and more citations (Gottfredson 1978, Lindsey 1978, PalaciosHuerta and Volij 2004. This creates circular causality: Articles receive more citations because they appear in higher-prestige journals, and journals gain prestige because they publish articles that receive more citations.…”