Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication

Abstract

Expert disagreement is pervasive in clinical decision making and collective adjudication is a useful approach for resolving divergent assessments. Prior work shows that expert disagreement can arise due to diverse factors including expert background, the quality and presentation of data, and guideline clarity. In this work, we study how these factors predict initial discrepancies in the context of medical time series analysis, examining why certain disagreements persist after adjudication, and how adjudication impacts clinical decisions. Results from a case study with 36 experts and 4,543 adjudicated cases in a sleep stage classification task show that these factors contribute to both initial disagreement and resolvability, each in their own unique way. We provide evidence suggesting that structured adjudication can lead to significant revisions in treatment-relevant clinical parameters. Our work demonstrates how structured adjudication can support consensus and facilitate a deep understanding of expert disagreement in medical data analysis.

A Method for Computing Conceptual Distances between Medical Recommendations: Experiments in Modeling Medical Disagreement

“…Our results show that it is possible to build computational models of "near-peer" disagreement. Additionally, they provide support for the empirical observations of disagreement adjudication among medical experts [34,35], where the authors observe that the differences in experts' backgrounds increase the degree of disagreement.…”

“…Once medical experts express their disagreements, what happens next? Observations from disagreement adjudication are analyzed in [34,35], where the authors observe (among other things) that the differences in experts' backgrounds increase the degree of disagreement.…”

Best practices for authors of healthcare-related artificial intelligence manuscripts

“…If labels are assigned by human experts, describe methods in detail. Describe any efforts to quantify, and mitigate, intra-and inter-observer labeling differences 5 . Also, describe how closely the temporal alignment of the labels relates to the data segments being assigned.…”

Difficulty-Aware Glaucoma Classification with Multi-rater Consensus Modeling

“…Glaucoma images, in fact medical images in general, are usually labeled by multiple experts independently, so as to avoid the subjective bias or potential labeling noise of each rater resulted by different levels of expertise, negligence of subtle symptoms, quality of images, etc [13]. The final ground-truth label then can be obtained by fusing individual labels using majority vote, average or other fusion strategies [13].…”

“…Glaucoma images, in fact medical images in general, are usually labeled by multiple experts independently, so as to avoid the subjective bias or potential labeling noise of each rater resulted by different levels of expertise, negligence of subtle symptoms, quality of images, etc [13]. The final ground-truth label then can be obtained by fusing individual labels using majority vote, average or other fusion strategies [13]. However, at model training stage, only the final ground-truth label is utilized to train the model and those intermediate labels generated by individual raters are neglected, which contain important information regarding the gradeability or difficulty levels of the images.…”