Reliability in evaluator-based tests: using simulation-constructed models to determine contextually relevant agreement thresholds

Abstract

BackgroundIndices of inter-evaluator reliability are used in many fields such as computational linguistics, psychology, and medical science; however, the interpretation of resulting values and determination of appropriate thresholds lack context and are often guided only by arbitrary “rules of thumb” or simply not addressed at all. Our goal for this work was to develop a method for determining the relationship between inter-evaluator agreement and error to facilitate meaningful interpretation of values, thresholds, and reliability.MethodsThree expert human evaluators completed a video analysis task, and averaged their results together to create a reference dataset of 300 time measurements. We simulated unique combinations of systematic error and random error onto the reference dataset to generate 4900 new hypothetical evaluators (each with 300 time measurements). The systematic errors and random errors made by the hypothetical evaluator population were approximated as the mean and variance of a normally-distributed error signal. Calculating the error (using percent error) and inter-evaluator agreement (using Krippendorff’s alpha) between each hypothetical evaluator and the reference dataset allowed us to establish a mathematical model and value envelope of the worst possible percent error for any given amount of agreement.ResultsWe used the relationship between inter-evaluator agreement and error to make an informed judgment of an acceptable threshold for Krippendorff’s alpha within the context of our specific test. To demonstrate the utility of our modeling approach, we calculated the percent error and Krippendorff’s alpha between the reference dataset and a new cohort of trained human evaluators and used our contextually-derived Krippendorff’s alpha threshold as a gauge of evaluator quality. Although all evaluators had relatively high agreement (> 0.9) compared to the rule of thumb (0.8), our agreement threshold permitted evaluators with low error, while rejecting one evaluator with relatively high error.ConclusionsWe found that our approach established threshold values of reliability, within the context of our evaluation criteria, that were far less permissive than the typically accepted “rule of thumb” cutoff for Krippendorff’s alpha. This procedure provides a less arbitrary method for determining a reliability threshold and can be tailored to work within the context of any reliability index.

Revisiting reopened bugs in open source software systems

“…We computed the inter-coder agreement using Krippendorff's α (Artstein and Poesio, 2008;Hayes and Krippendorff, 2007). The inter-coder agreement for our study is 0.87 which concludes that our qualitative analysis is reliable as prior studies use α 0.8 as an indicator of reliable agreement (Li et al, 2020;Beckler et al, 2018;Webb et al, 2020;Vassallo et al, 2020;Scoccia and Autili, 2020).…”

Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors

“…Organisms instinctively make strategic decisions about how to spend their resources in relationship to the payoff from their actions. Humans are subject to these same instincts (72)(73)(74)(75), and PEP provides insight into resource allocation decisions made during sensory discrimination tasks (49,50). With TMR-motor only, pSD made discrimination decisions slightly above chance.…”

“…The metrics focus on essential sensory-motor features of limb function such as visual attention, cognitive demand, fine motor dexterity, and ownership while also being clinically and real-world relevant. Each metric was validated in their foundational fields of psychophysics, mathematical theory, cognition/perception, visuomotor behavior, and psychometrics (23,31,(37)(38)(39)(40)(41)(42)(43)(44)(45)(46)(47)(48)(49)(50)(51)(52)(53)(54).…”

The relationship of polarity of post-publication peer review to citation count

“…Krippendorff's alpha of the polarity of the PPPRs and the similarities between pre-publication peer reviews and PPPRs were calculated using a software developed by Freelon (2013), and the alpha values were 0.887 and 0.941. Both alpha values were greater than 0.8, which demonstrated acceptable interrater reliability between the judgments of the two coders (Beckler et al , 2018).…”