The inter-rater reliability of the story retell procedure

Abstract

2001) have presented the story retell procedure (SRP) as an efficient means of assessing discourse in adults with aphasia, in part because it provides reliable, valid, and sensitive indices of performance without the need for time-consuming transcription of language samples. Aims: The purpose of this study was to demonstrate that the SRP, when scored without transcription by judges with minimal training, produces a reliable measure of information transfer.Methods & Procedures: Four judges who had not used the SRP previously scored audiorecorded language samples, produced by four subjects with aphasia and eleven normal subjects, for percent information units per minute (%IU/Min). Outcomes & Results:The results demonstrate that the SRP has high inter-rater reliability. Reliability coefficients ranged from .89 to .995, and the standard error of measurement associated with inter-rater scoring error ranged from .59 to 1.42 %IU/Min. Point-to-point reliability in scoring individual information units ranged from 85±95% and averaged 91% for both subject groups. Conclusions: The SRP is a potentially useful tool for quantifying connected language behaviour, and may be particularly valuable in clinical and research settings where economy of assessment procedures is essential.

Effects of Examiner Error on Neuropsychological Test Results in a Multi-site Study

“…This finding is consistent with a previous study that found Wechsler scales that require subjective scoring are prone to examiner error even though a structured feedback system is in place (Connor & Woodall, 1983). Past studies that have assessed examiner scoring errors on individual tests of prose passages similar to Wechsler Logical Memory have found generally good inter-rater reliability (Hula et al, 2003;Sullivan, 2000;Woloszyn et al, 1993). These studies employed few raters and/or prose passages that were not derived from actual clinical patients.…”

“…Woloszyn, Grob-Murphy, Wetzel, and Fisher (1993) found almost perfect agreement between two examiners on both Logical Memory and Visual Reproduction subtest scoring of the Wechsler Memory Scale -Revised (Wechsler, 1987). Hula, McNeil, Doyle, Rubinsky, and Fossett (2003) reported high inter-rater reliability on a story retell procedure scored by four individuals with varying experience in aphasia and language assessment. Good inter-rater reliability was found by Sullivan (1996) in a study that compared two test conditions (scoring story transcripts only versus listening to, recording, and scoring) non-clinical prose responses.…”

Objective measurement of fluency in natural language production: A dynamic systems approach

“…Our analysis of CIU follows that of Nicholas and Brookshire (1993), and the temporal designation (s/CIU) was introduced to make it comparable to lexical decision. It is of course inter-changeable with minutes per CIU as used by Hula, McNeil, Doyle, Rubinsky, and Fossett (2003).…”

Reliability of brain computer interface language sample transcription procedures

“…Studies that report results based on language samples typically report the reliability of the transcription procedures that have been operationalized for the protocol [22][23]. Highly consistent transcription of language samples indicates that the language or performance measures are both reliable and valid.…”