Sources of listener disagreement in voice quality assessment

Abstract

Traditional interval or ordinal rating scale protocols appear to be poorly suited to measuring vocal quality. To investigate why this might be so, listeners were asked to classify pathological voices as having or not having different voice qualities. It was reasoned that this simple task would allow listeners to focus on the kind of quality a voice had, rather than how much of a quality it possessed, and thus might provide evidence for the validity of traditional vocal qualities. In experiment 1, listeners judged whether natural pathological voice samples were or were not primarily breathy and rough. Listener agreement in both tasks was above chance, but listeners agreed poorly that individual voices belonged in particular perceptual classes. To determine whether these results reflect listeners' difficulty agreeing about single perceptual attributes of complex stimuli, listeners in experiment 2 classified natural pathological voices and synthetic stimuli ͑varying in f 0 only͒ as low pitched or not low pitched. If disagreements derive from difficulties dividing an auditory continuum consistently, then patterns of agreement should be similar for both kinds of stimuli. In fact, listener agreement was significantly better for the synthetic stimuli than for the natural voices. Difficulty isolating single perceptual dimensions of complex stimuli thus appears to be one reason why traditional unidimensional rating protocols are unsuited to measuring pathologic voice quality. Listeners did agree that a few aphonic voices were breathy, and that a few voices with prominent vocal fry and/or interharmonics were rough. These few cases of agreement may have occurred because the acoustic characteristics of the voices in question corresponded to the limiting case of the quality being judged. Values of f 0 that generated listener agreement in experiment 2 were more extreme for natural than for synthetic stimuli, consistent with this interpretation.

Reliability of perceptions of voice quality: evidence from a problem asthma clinic population

“….pathologic voice quality assessment using traditional perceptual labels [viz 'breathy', 'rough', etc] is not generally useful'. 16 The GRBAS scale is a perceptual rating scale widely reported in voice research. 17 Using a four-point scale (where zero ¼ normal and three ¼ severe), this system characterises the voice according to grade (i.e.…”

The Effect of the Speech Task Characteristics on Perceptual Judgment of Mild to Moderate Dysphonia: A Methodological Study

“…These findings represent a kind of lack of experience regarding varied clinical practice. We may also postulate that other factors may explain these results and include the mental or psychic condition of the judge, potential fatigue or attention lapse at the moment of the evaluations, or the voice types listened in their daily consultation before the beginning of the assessment [13, 32]. …”

Hypernasal Speech Is Perceived as More Monotonous than Typical Speech

“…However, listeners may base their auditory-perceptual evaluations on different individual internal standards, which can detract from interrater reliability [36]. When multiple disordered features are present in a complex case, it becomes more difficult for listeners to isolate individual auditory-perceptual features with any accuracy [37]. …”