Holistic assessment of narrative quality: A social validation study

Abstract

Background: There is a tradition of social validation studies in the adult clinical treatment literature that has not yet been extended to narrative evaluation. This social validation process begins with lay listener holistic judgements of narrative quality for non-brain-injured narrators, and compares these judgements to clinician-researchers' assessments of the same narratives. The resulting narratives rated for quality reflect the premorbid range of skills in clinical populations, and overall narrative performance. Narratives whose quality has been assessed holistically by laypersons can be further examined for the qualitative and quantitative dimensions that characterise them. Aims: This study explores the social validity of clinical narrative assessment. Specifically, it examines the degree to which clinicians' holistic and analytic assessments of narrative quality parallel holistic narrative quality judgements of a lay audience, for a set of narratives produced by non-brain-injured narrators. It also examines the characteristics of narratives along a continuum of quality. Methods & Procedures: A total of 14 African American and 15 Caucasian lay individuals holistically rated the quality of personal narratives produced by ethnicity-matched groups of 37 African-American and 34 Caucasian non-brain-injured narrators, respectively. Five clinical raters holistically rated all narratives. Additionally, lay raters were asked for their intuitions about what contributes to narrative quality. They also ranked six researcherspecified dimensions for their importance in determining narrative quality. High-rated and low-rated narratives were compared on dimensions of length, content, and narrative structure.Outcomes & Results: Agreement between lay and clinical rater groups was found for a subset of high-rated and low-rated stories, which constituted one-quarter to one-third of the total narratives. A similar level of agreement was found for male and female lay raters. Ratergroup disagreements at the extremes of narrative quality were rare. Certain of the lay raters' characterisations of good and poor narratives corresponded to narrative dimensions discussed in the literature and defined by the researchers. High-rated stories were longer, more conventionally structured, and different in content from low-quality stories. Conclusions: For personal narratives at the extremes of perceived quality (high and low), evidence for social validity in narrative assessment is manifested both at the holistic assessment level, and at the level of the dimensions thought to contribute to narrative quality. The results contribute to our understanding of the normal range of narrative performance.

Women's Initial Experiences and up to a Year in Retrospect of Their Life Situation With a Confirmed Takotsubo Syndrome Diagnosis: A Single Case Study Literature Review

“…Similarly, the use of a well-proven quality assessment form (Streit Olness et al, 2005) contributed to ensuring the social validity of the individual clinical statements. During the quality assessment and data analysis processes, the research team was conscious of the group's preunderstanding and made no assumptions without reviewing and analyzing the studies until consensus was reached.…”

“…In accordance with Streit Olness et al (2005), all articles were carefully read to assess the quality of the social validity of the individual clinical statements. Specifically, the focus was on evaluating researchers and clinicians' analytic and holistic assessment of the statements' quality, and overall, the characteristics of patient statements along a continuum of quality.…”

The effect of workshop training on rater variability in children’s oral narrative assessment

“…SS was a story from an older child and showed more complexity than WBB, as illustrated by more and longer utterances, more diverse vocabulary, and higher MISL scores. Previous research comparing ratings of narrative quality found agreement was highest at the ‘polar extremes’ of the rating scale (Streit Olness et al, 2005). It is possible that, since SS was a more complex and developed story, it was clearer for all raters, regardless of experience, to identify strengths and score consistently.…”