Correcting for rater bias in scores on a continuous scale, with application to breast density

Abstract

Existing literature on inter-rater reliability focuses on quantifying the disagreement between raters. In this paper, we introduce a method to correct for inter-rater disagreement (or observer bias), where raters are assigning scores on a continuous scale. To do this, we propose a two-stage approach. In the first stage, we standardise the distributions of rater scores to account for each rater's subjective interpretation of the continuous scale. In the second stage, we correct for case-mix differences between raters by exploiting pairwise information where two raters have read the same entity on a case. We illustrate the use of our procedure on clinicians' visual assessments of breast density (a risk factor for breast cancer). After applying our procedure, 229 out of 1398 women who were originally classified as high density were re-classified as non-high density, and 382 out of 12 348 women were re-classified from non-high to high density. A simulation study also demonstrates good performance of the proposed method over a range of scenarios.

The TOMMY trial: a comparison of TOMosynthesis with digital MammographY in the UK NHS Breast Screening Programme – a multicentre retrospective reading study comparing the diagnostic performance of digital breast tomosynthesis and digital mammography with digital mammography alone

“…Observer measurement of breast density has been shown in other studies to suffer from interobserver variability 123,124,131,132 and a recent paper has attempted to provide a correction for differences between observers. 133 From our results it is clear that observers are able to discriminate different densities in subjects with low automated breast densities. However, when examining the histograms, shown in Figure 14, of the area-based measurements from both human observers and software analysis, there is clearly a difference in the distribution of scores.…”

Mammographic density adds accuracy to both the Tyrer-Cuzick and Gail breast cancer risk models in a prospective UK screening cohort

“…More follow-up is needed to help to address this issue. Thirdly, the visually assessed score required human judgement, which might make it unreliable for routine use in a screening program [ 31 ], although the same applies to BI-RADS density. Exploration of automated methods is ongoing in a subset of the cohort.…”

Qualitative Versus Quantitative Mammographic Breast Density Assessment: Applications for the US and Abroad

“…VAS has been used in large clinical studies [30,31,32,33,34] and is considered preferable to some of the thresholding-based methods (described below) as it is less laborious and does not require specific reader training.…”