The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?

Abstract

The prevalence of faulty citations impedes the growth of scientific knowledge. Faulty citations include omissions of relevant papers, incorrect references, and quotation errors that misreport findings. We discuss key studies in these areas. We then examine citations to "Estimating nonresponse bias in mail surveys," one of the most frequently cited papers from the Journal of Marketing Research, to illustrate these issues. This paper is especially useful in testing for quotation errors because it provides specific operational recommendations on adjusting for nonresponse bias; therefore, it allows us to determine whether the citing papers properly used the findings. By any number of measures, those doing survey research fail to cite this paper and, presumably, make inadequate adjustments for nonresponse bias. Furthermore, even when the paper was cited, 49 of the 50 studies that we examined reported its findings improperly. The inappropriate use of statistical-significance testing led researchers to conclude that nonresponse bias was not present in 76 percent of the studies in our sample. Only one of the studies in the sample made any adjustment for it. Judging from the original paper, we estimate that the study researchers should have predicted nonresponse bias and adjusted for 148 variables. In this case, the faulty citations seem to have arisen either because the authors did not read the original paper or because they did not fully understand its implications. To address the problem of omissions, we recommend that journals include a section on their websites to list all relevant papers that have been overlooked and show how the omitted paper relates to the published paper. In general, authors should routinely verify the accuracy of their sources by reading the cited papers. For substantive findings, they should attempt to contact the authors for confirmation or clarification of the results and methods. This would also provide them with the opportunity to enquire about other relevant references. Journal editors should require that authors sign statements that they have read the cited papers and, when appropriate, have attempted to verify the citations. The prevalence of faulty citations impedes the growth of scientific knowledge. Faulty citations include omissions of relevant papers, incorrect references, and quotation errors that misreport findings. We discuss key studies in these areas. We then examine citations to "Estimating nonresponse bias in mail surveys," one of the most frequently cited papers from the Journal of Marketing Research, to illustrate these issues. This paper is especially useful in testing for quotation errors because it provides specific operational recommendations on adjusting for nonresponse bias; therefore, it allows us to determine whether the citing papers properly used the findings. By any number of measures, those doing survey research fail to cite this paper and, presumably, make inadequate adjustments for nonresponse bias. Furthermore, even when the paper was cited, 49 of th...

Factors enhancing the choice of higher resource commitment entry modes in international new ventures

“…Gendall (2000) and Wright and Armstrong (2008) state one of the most effective and reliable factors that could minimize non-response bias is to achieve a high response rate. Another way to ensure that the probability of incurring non-response is low is comparing the responses of early and late respondents (Armstrong and Overton 1977).…”

The Ombudsman: Are Top Executives Paid Enough? An Evidence-Based Review

“…Academic papers frequently misstate findings in their reviews of prior research (Wright and Armstrong 2008). To address this, we contacted authors whose findings we summarized in substantive ways.…”

Academic sell-out: How an obsession with metrics and rankings is damaging academia

“…For example, Simkin and Roychowdhury (2003) used stochastic modeling of the citation process to estimate that only about 20% of academics who cite articles actually read the original. Further, Wright and Armstrong (2008) analyzed citations to one of the most cited articles in marketing (Armstrong & Overton, 1977) and found that in 49 of the 50 studies they examined, the findings were reported incorrectly, indicating that the citers did not read the original paper (or did not understand it properly).…”