On Jones et al.’s method for extending Bland-Altman plots to limits of agreement with the mean for multiple observers

Abstract

Background To assess the agreement of continuous measurements between a number of observers, Jones et al. introduced limits of agreement with the mean (LOAM) for multiple observers, representing how much an individual observer can deviate from the mean measurement of all observers. Besides the graphical visualisation of LOAM, suggested by Jones et al., it is desirable to supply LOAM with confidence intervals and to extend the method to the case of multiple measurements per observer.Methods We reformulate LOAM under the assumption the measurements follow an additive two-way random effects model. Assuming this model, we provide estimates and confidence intervals for the proposed LOAM. Further, this approach is easily extended to the case of multiple measurements per observer.Results The proposed method is applied on two data sets to illustrate its use. Specifically, we consider agreement between measurements regarding tumour size and aortic diameter. For the latter study, three measurement methods are considered. Conclusions The proposed LOAM and the associated confidence intervals are useful for assessing agreement between continuous measurements.

Development and Application of a Web-Based Platform for Assessment of Observer Performance in Medical Imaging

“…The sentence "...a total of 1350 measurements and 450 caliper placements for each of the three methods…" should have read "...a total of 1800 measurements and 600 caliper placements for each of the three methods". In addition, in 2020, Christensen et al have provided an extension of this model with interobserver variance incorporated, and thus reproducibility 66 . In this update, limits of agreement can be defined as how much a given observer's measurement may plausibly deviate from the mean of all observers' measurements on the specific subject .…”

“…We reformulated the Jones LOAM under an additive two-way random effects model described in a paper by Christensen et al 66 . In this model, the total variation present in a set of measurements by different observers is partitioned into components attributable to different sources of variation: the inter-subject (σ A , i.e., variation between the true values for subjects), inter-observer (σ B , i.e., varying bias between observers where observers consistently measure high or low), and residual variance (σ E ).…”

“…Abbreviation: OTO = outer to outer; LTL = leading to leading edge; ITI = inner to inner. Reprinted from BMC Medical Research Methodology 2020 49,66 , copyright, by permission of Springer.…”