The Americleft Speech Project: A Training and Reliability Study

Abstract

Objective To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores. Design The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech–Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2. Participants The participants were speech-language pathologists from the Americleft Speech Project. Results In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability results for the two studies indicated lower scores for Study 2 compared with Study 1. However, this appeared to be an artifact of the kappa statistic that occurred due to insufficient variability in the reliability samples for Study 2. When percent agreement scores were also calculated, the ratings appeared similar across Study 1 and Study 2. Conclusion The findings of this study suggested that improvements in interrater reliability could be obtained following a program of systematic training. However, improvements were not uniform across all parameters. Acceptable levels of reliability were achieved for those parameters most important for evaluation of velopharyngeal function.

Influência do treinamento dos avaliadores no julgamento perceptivo da hipernasalidade

“…Several programs of training have been documented in literature, mostly with positive results. A large number was used in the analysis of vocal aspects, such as roughness and breathiness (14,15,17,18,20) , and only a few in the analysis of nasality (6,7,16,19) . One of the more effective training strategies mentioned in literature is the use of the reference-matching, as the one that used in this study (20) .…”

“…Some authors believe that listeners should be trained prior to the assessment of hypernasality to adjust their internal scales, i.e., they should classify speech samples with different degrees of nasality until they reach an agreement in their own judgments (13) . Studies contemplating the auditory perceptual assessment of voice quality have shown that the use of anchors, prior listener training, and experience of the listeners favor the reliability of results (7,(14)(15)(16)(17)(18)(19) . Regarding the vocal aspects, multiple authors have demonstrated the efficacy in both the listener training and the use of anchors in the perceptual assessment of dysphonia symptoms.…”

“…In a recent study (19) , experts defended these same strategies stating that, although the perceptual assessment of cleft speech symptoms presents limitations and great variability, prior listener training can be used to achieve acceptable levels of reliability, regardless of the level of listener experience. These authors described the reliability levels for the speech assessment of two perceptual assessment protocols (Cleft Audit Protocol for Speech-Augmented, CAPS-A, and Cleft Audit Protocol for Speech-Augmented-Americleft Modification, CAPS-A-AM) developed to assess speech results in intercenter collaborative studies and investigated the effect of training on the agreement among different listeners.…”

Hypernasal Speech Is Perceived as More Monotonous than Typical Speech

“…The variability has been attributed to the types of rating scales used [41] or to listeners’ experience [42, 43]. Studies about listener training have shown improved interrater agreement [44-46]. However, it is largely unknown how listeners perform when there are co-occurring pathologies in more than one perceptual dimension, which is often the case in clinical practice.…”

Influência do uso de amostras de referência no julgamento perceptivo-auditivo da oclusiva glotal

“…Estudos (PAMPLONA et al, 2012;BESSELL et al, 2013) Apesar de existirem métodos objetivos para a avaliação da fala como a avaliação aerodinâmica e análise acústica o julgamento perceptivo-auditivo é o método de avaliação considerado como "padrão ouro" para a caracterização das alterações de fala (GERRATT et al, 1993;KREIMAN et al, 1993;KENT, 1996;KUEHN;MOLLER, 2000;KUMMER et al, 2012;CHAPMAN et al, 2015). A avaliação perceptiva das alterações de fala, incluindo as AC, pode sofrer influência de vários fatores (SANTELMAN; SUSSMAN; CHAPMAN, 1999;GOOCH et al, 2001;HOWARD;HESELWOOD, 2002;HENNINGSSON et al, 2008), dentre estes destacam-se a experiência do avaliador em realizar o julgamento perceptivo-auditivo destas produções (GOOCH et al, 2001;OLSSON, 2004;BRUNNEGARD;LOHMANDER, 2007), bem como a seleção e qualidade das amostras de fala utilizadas durante o julgamento.…”