Lems of Analyst and Observer Agreement in Naturalistic Narrative Data

Abstract

The use of naturalistic narrative data is increasing but without increasing methodological rigor. Assumptions underlying such methods prohibit an investigator from interrupting the natural habitat by use of such standard laboratory procedures as equal observation intervals for all subjects, equal behavior units, or other artificial equalization of the behavioral data stream. Data analytic techniques commonly used in standard laboratory research present problems in the analysis of naturalistic data of this type inasmuch as most assume equal observation intervals, equal unit intervals, or some other fundamental character of equality. These problems are particularly acute with respect to estimates of analyst and observer agreement. This paper addresses these problems and discusses the advantages and disadvantages of several possible solutions.The collection and analysis of naturalistic observational data present some unique problems in determining observer and coder reliability and agreement. A number of these problems, especially as they relate to category observation schedules, have been extensively reviewed have also addressed these problems as they relate to ethnographic research. However, the particular problems surrounding determination of observer and analyst agreement in narrative and observational records have not been well explored. Furthermore, the use of naturalistic narrative data is increasing without concomitant increase in methodological rigor.The purposes of this paper are (a) to describe problems that researchers encounter when working in this area, (b) to identify various solutions that have been used, (c) to report results of some recent studies of observer and analyst agreement, (d) to propose alternate strategies for calculating analyst agreement, and (e) to compare and contrast this strategy with others previously used. The focus of this paper is on interobserver and interanalyst agreement. The issues raised and the methods discussed, however, would be equally applicable to intraobserver and intraanalyst agreement. PROBLEMS IN DETERMINING OBSERVER AND ANALYST AGREEMENTThe methodology associated with narrative observation provides a framework for considering the unique difficulties encountered with this type of observation in estimating Requests for reprints should be sent to M. M. Scott,

The Relationship Between Students’ Reading Orientations and their Strategic Activity During a Collaborative Reading Task

“…Interrater agreement was calculated for each strategy type and participation role using a variation of Cohen's kappa appropriate for frequency data (Scott & Hattfield, 1985). Cohen's kappas ranged from .85 to 1.00.…”

Analyzing Cultural Models in Adolescent Accounts of Romantic Relationships

“…A TU is the smallest unit that communicates a complete idea and typically varies in size from a clause to three sentences. Four judges were trained to a criterion reliability of .76 (81% agreement; Scott & Hatfield, 1985) on four standard segmented transcripts. Following training two judges segmented each transcribed interview independently and a third judge reviewed and reconciled any differences between the marks.…”

Individual and Environmental Determinants of Engagement in Autism

“…The AUs were unitized by a rater who was unaware of the diagnosis of the children. A description of unitizer training is provided in Appendix C. The Scott and Hatfield (1985) method was used to determine the percentage of inter-rater agreement. This method allows for the duration of behavior as part of the analysis (see Appendix C).…”