Rater Methodology for Stroboscopy: A Systematic Review

Abstract

SUMMARY Objectives Laryngeal endoscopy with stroboscopy (LES) remains the clinical gold standard for assessing vocal fold function. LES is used to evaluate the efficacy of voice treatments in research studies and clinical practice. LES as a voice treatment outcome tool is only as good as the clinician interpreting the recordings. Research using LES as a treatment outcome measure should be evaluated based on rater methodology and reliability. The purpose of this literature review was to evaluate the rater-related methodology from studies that use stroboscopic findings as voice treatment outcome measures. Study Design Systematic Literature Review Methods Computerized journal databases were searched for relevant articles using terms: stroboscopy and treatment. Eligible articles were categorized and evaluated for the use of rater-related methodology, reporting of number of raters, types of raters, blinding and rater reliability. Results Of the 738 articles reviewed, 80 articles met inclusion criteria. More than one-third of the studies included in the review did not report the number of raters who participated in the study. Eleven studies reported results of rater reliability analysis with only two studies reporting good inter- and intra-rater reliability. Conclusion The comparability and use of results from treatment studies that employ LES are limited by a lack of rigor in rater methodology and variable, mostly poor, inter- and intra-rater reliability. To improve our ability to evaluate and use the findings from voice treatment studies that employ LES features as outcome measures, greater consistency of reporting rater methodology characteristics across studies and improved rater reliability is needed.

A Preliminary Quantitative Comparison of Vibratory Amplitude Using Rigid and Flexible Stroboscopic Assessment

“…This subjectivity can be problematic for the reliability of such ratings when used with stroboscopic evaluation. [14][15][16][17] For improved reliability, objective methods for the assessment of vocal fold amplitude from stroboscopic examinations are needed.…”

Laryngeal examination in thyroid and parathyroid surgery: An American Head and Neck Society consensus statement

“…The technique has certain limitations and should be avoided in cases in which impaired periodicity may make synchronization with the strobe light problematic . Furthermore, its interpretation is subjective; there is not an established measurement tool with demonstrated interrater reliability, internal consistency, and discriminant validity . Thus, comparisons or pooling data at a national and international level may be difficult because of heterogeneity in the collection and measurement systems as well as the diagnostic terminology.…”

“…34,35,39 Furthermore, its interpretation is subjective; there is not an established measurement tool with demonstrated interrater reliability, internal consistency, and discriminant validity. 34,40 Thus, comparisons or pooling data at a national and international level may be difficult because of heterogeneity in the collection and measurement systems as well as the diagnostic terminology. Stroboscopy is not the only technique available for a detailed evaluation of laryngeal movement and pathology; others include electroglottography, videokymography, and high-speed digital imaging, and some techniques may be complementary.…”

Does the accuracy of case history affect interpretation of videolaryngostroboscopic exams?

“…The most important implication of this study is that because case history affected the perceived severity of some VLS parameters, ratings should be performed blinded when used to measure treatment outcomes. Yet, rater blinding was reported in only 31% of studies using VLS as a primary treatment outcome measure . However, it is difficult to reconcile the advantages of an accurate preliminary diagnostic hypothesis in detecting VLS abnormalities with the disadvantages of an inaccurate hypothesis during interpretation of an exam in a clinical setting when the purpose of such an exam is to assess laryngeal function and structure, render a diagnosis, and make treatment recommendations.…”

“…Yet, rater blinding was reported in only 31% of studies using VLS as a primary treatment outcome measure. 33 However, it is difficult to reconcile the advantages of an accurate preliminary diagnostic hypothesis in detecting VLS abnormalities with the disadvantages of an inaccurate hypothesis during interpretation of an exam in a clinical setting when the purpose of such an exam is to assess laryngeal function and structure, render a diagnosis, and make treatment recommendations. Thus, it is important that clinicians and researchers are aware of how cognitive biases affect visual-perceptual judgments because this may affect how VLS results are interpreted and how future investigations should be designed.…”