Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

Abstract

As a method specifically intended for the study of messages, content analysis is fundamental to mass communication research. Intercoder reliability, more specifically termed intercoder agreement, is a measure of the extent to which independent judges make the same coding decisions in evaluating the characteristics of messages, and is at the heart of this method. Yet there are few standard and accessible guidelines available regarding the appropriate procedures to use to assess and report intercoder reliability, or software tools to calculate it. As a result, it seems likely that there is little consistency in how this critical element of content analysis is assessed and reported in published mass communication studies. Following a review of relevant concepts, indices, and tools, a content analysis of 200 studies utilizing content analysis published in the communication literature between 1994 and 1998 is used to characterize practices in the field. The results demonstrate that mass communication researchers often fail to assess (or at least report) intercoder reliability and often rely on percent agreement, an overly liberal index. Based on the review and these results, concrete guidelines are offered regarding procedures for assessment and reporting of this important aspect of content analysis.T he study of communication is interdisciplinary, sharing topics, literatures, expertise, and research methods with many academic fields and disciplines. But one method, content analysis, is specifically appropriate and necessary for (arguably) the central work of communication scholars, in particular those who study mass communication: the analysis of messages. Given that content analysis is fundamental to communication research (and thus theory), it would be logical to expect researchers in communication to be among the most, if not the most, proficient and rigorous in their use of this method.

Evolution of newspaper coverage of water issues in Australia during 1843–2011

“…Manual coding content analysis was chosen to observe the change of published information about water management issues over time as we believe that important elements of arguments are often implied rather than explicit in statements, and human coders are more alert to those latent content beneath the manifest content (Lombard et al 2002;Howland et al 2006). This approach enables data patterns to be determined from unstructured information, and it provides a powerful tool for tracking changes in media coverage (Higuchi 2004;Kirilenko et al 2012).…”

A Comparison of Phone-Based and On-Site Assessment of Fidelity for Assertive Community Treatment in Indiana

“…Calculations for all ICCs followed Shrout and Fleiss (13) Model 2 and used two-way random effects ANOVA with absolute agreement. ICCs above .90 are very good, above .80 are acceptable, and above .70 are adequate for exploratory research (14). …”

Reliability in Content Analysis: Some Common Misconceptions and Recommendations

“…As already mentioned, Lombard et al (2002) applied the following criterion for accepting content analysis findings as sufficiently reliable:   .70, otherwise %-agreement  .90 (p. 596). They take  as a conservative index and %-agreement as a liberal one, presumably convinced that the truth lies somewhere between these two.…”

“…Amplifying Neuendorf (2002, p. 163) who merely quotes a concern expressed elsewhere about the appropriateness of using different coders for coding different but overlapping sets of units, Lombard et al (2002) make it a point of recommending against this attractive possibility (p. 602) -without justification, however. I can imagine three: (a) Potter and Levine-Donnerstein (1999) argue that the overlap needs to be large enough, which is correct.…”

“…

AbstractIn a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data.

…”