Guidelines for Establishing Reliability When Coding Narrative Data

Abstract

The use of quantitative, qualitative, and mixed methods approaches has been foundational to research on emerging adulthood, yet there remain many unresolved methodological issues pertaining to how to handle qualitative data. The purpose of this article is to review best practices for coding and establishing reliability when working with narrative data. In doing so, we highlight how establishing reliability must be seen as an evolving process, rather than simply a focus on the end product. The review is divided into three broad sections. In the first section, we discuss relatively more quantitatively focused methods of coding and establishing reliability, whereas in the second section we discuss relatively more qualitatively focused methods. In the final section, we provide recommendations for researchers interested in coding narrative and other types of open-ended data. This article is intended to serve as an essential resource for researchers working on a variety of topics related to emerging adulthood and beyond.

Research Methods for Studying Narrative Identity: A Primer

“…These scores are then compared statistically to determine whether inter-rater reliability has been achieved. There are different approaches to calculating inter-rater reliability, depending on the nature of the coding systems (see Syed & Nelson 2015). Raters rarely achieve appropriate inter-rater reliability on the first attempt, in which case they should discuss disagreements, come to consensus for those data points, refine the coding system if necessary, and then attempt another set of coding independently.…”

“…This intensive process is described by Syed and Nelson (2015; see also McAdams et al, 2008). In addition, applying an existing coding system to a new dataset is not always straightforward.…”

“…This training phase is crucial to the scientific soundness of the coding, which will ultimately produce the quantitative representation of the narrative data. As Syed and Nelson (2015) point out, inter-rater reliability is best conceived of as a process and they outline the many decision points in this process. While a set of statistics will represent this procedure in any report of the study, inter-rater reliability informs every step of the coding process.…”

‘It was me answering my own questions’: Experiences of method of levels therapy amongst people with first‐episode psychosis

“…• A 'gold standard/master coder' approach to coding the dataset was used, where a second rater's codes (HP) were used to increase the credibility of the coding schedule, but only RG codes contributed to the final analysis (Syed & Nelson 2015). RG coded a randomly selected transcript using the initial coding schedule.…”

The Association of Strategy Use and Concrete-Operational Thinking in Primary School

“…Two external raters were invited for coder training. Following a three-step procedure proposed by Syed and Nelson (2015), a coding manual was introduced, randomly drawn sample data were provided for practice, and the coders' questions were clarified. The coding scheme contains mental operations used in the problem-solving process (see Figure 2).…”