Identifying Sources of Difference in Reliability in Content Analysis

Abstract

This paper reports on a case study which identifies and illustrates sources of difference in agreement in relation to reliability in a context of quantitative content analysis of a transcript of an online asynchronous discussion (OAD). Transcripts of 10 students in a month-long online asynchronous discussion were coded by two coders using an instrument with two categories, five processes, and 19 indicators of Problem Formulation and Resolution (PFR). Sources of difference were identified in relation to: coders; tasks; and students. Reliability values were calculated at the levels of categories, processes, and indicators. At the most detailed level of coding on the basis of the indicator, findings revealed that the overall level of reliability between coders was .591 when measured with Cohen's kappa. The difference between tasks at the same level ranged from .349 to .664, and the difference between participants ranged from .390 to .907. Implications for training and research are discussed.

Collaboration indices for monitoring potential problems in online small groups / Indicateurs de collaboration pour diagnostiquer des problèmes potentiels dans de petits groupes en ligne

“…As discussed in the introduction of this paper, there are common issues of efficacy and reliability of coding written data in conducting a content analysis (Murphy & Ciszewska-Carr, 2005). Thus, evaluating collaboration process by employing a content analysis technique may not be feasible for instructors.…”

A Pilot Study of Problem Formulation and Resolution in an Online Problem-based Learning Course

“…A second issue concerns the suitability of the PFR instrument for measuring problem solving in a PBL context. Murphy and Ciszewska-Carr (2005) obtained a high level of inter-coder agreement (a kappa coefficient of 0.825 for the two categories of Problem Formulation and Problem Resolution), but they also used the instrument to analyze a discussion that was specifically structured to parallel the problem formulation and resolution process. The discussion was divided into eight tasks, each of which asked the learners to apply a step in the PFR model; e.g., the first task required participants to reflect on their initial knowledge of the problem and to post a message describing their understanding of the problem.…”

IRRODL Volume 7, Number 3

“…The first author coded the discussions for both Group 1 Case 2 and for Group 2 Case 2, while a graduate assistant (the third author) also coded Group 1 Case 2 and the second author coded the Group 2 Case 2 discussion. To code, we followed the data analysis processes outlined in Murphy and Ciszewska-Carr (2005), in which the authors advocate the paragraph as unit of analysis and recommend a three level analysis process: 1) first to code units at the level of the category (i.e., is this unit an example problem formulation/understanding or is it problem solving/ resolution? ); 2) next to re-code at the level of the process; and 3) to code a third time using the indicators.…”