Interobserver Reliability and Perceptual Ratings

Abstract

The purpose of this study was to examine the reliability of ratings of perceptual characteristics for 10 ataxic dysarthric subjects. The influence of the occurrence of "deviant" speech parameters on the calculation of reliability coefficients was also explored. Results indicated that overall interobserver agreement levels for minimally trained judges compared favorably to reliability coefficients reported in previous studies. Furthermore, levels of overall agreement were above levels of agreement expected on the basis of chance alone. In contrast to overall interobserver agreement, much lower levels of interobserver agreement were obtained when "occurrence Reliability" coefficients were calculated for deviant dimensions alone. However, occurrence reliability coefficients surpassed the level of agreement expected on the basis of chance alone for all subjects. Based on the results of this investigation, recommendations are made for modifying standard practices for obtaining interobserver reliability for perceptual ratings of speech characteristics.

The Redesign and Re-evaluation of an Internet-Based Telerehabilitation System for the Assessment of Dysarthria in Adults

“…Many researchers consider a PCA of above 80% to denote high reliability. 8,11,21 As the PCA for all of the rating scale groups in both the oromotor and perceptual assessments in both environments were above 93.75%, we feel confident that the inter-rater reliability for both environments was high.…”

“…Furthermore, the PCA for each of the groups of scales was above 88.54%, indicating high agreement (>80%) between the methods. 8,11,21 Thus, despite some low quadratic weighted κ statistics, the PEA and PCA values would indicate that telerehabilitation assessment of oromotor and perceptual speech characteristics is comparable to FTF assessment. The results from the analysis of the ASSIDS data were positive, with both the percentage sentence intelligibility and CER exceeding the predetermined clinical criteria described by the Hill et al 5 study.…”

“…With regard to percentage agreement, high reliability was determined using a criterion of 80% PCA. 8,11,21 Intrarater reliability using PEA was also examined for the diagnosis of type of dysarthria within each method. For the ASSIDS, inter-and intrarater reliability was explored using ICC.…”

Comparability of Perceptual Analysis of Speech Characteristics in Australian and Swedish Speakers with Multiple Sclerosis

“…[9][10][11] and in the present study], the lower the agreement achieved. Reported agreement values are around 0.80 B one scale value [6,9,10,18]. Darley et al [11] reported intra-judge reliability of 0.85 and inter-judge reliability of 0.84.…”

The place of perceptual analysis of dysarthria in the differential diagnosis of corticobasal degeneration and Parkinson's disease

“…If perceptual evaluation is to be considered an effective and reliable aid in diagnosis and classification, it becomes clinically relevant to determine the accuracy of this method when used in isolation. However, few studies have so far tried to diagnose specific dysarthria types with perceptual analysis alone [7,20,21]. Judges blinded to the medical diagnoses listened to audiotapes containing several types of dysarthria and were asked to derive the neurological disease or the dysarthria type for each patient: whatever the response procedure overall accurate identification rates were low, between 19 % and 56 % [20].…”