Bench-to-bedside review: The importance of the precision of the reference technique in method comparison studies – with specific reference to the measurement of cardiac output

Abstract

Bland-Altman analysis is used for assessing agreement between two measurements of the same clinical variable. In the field of cardiac output monitoring, its results, in terms of bias and limits of agreement, are often difficult to interpret, leading clinicians to use a cutoff of 30% in the percentage error in order to decide whether a new technique may be considered a good alternative. This percentage error of ± 30% arises from the assumption that the commonly used reference technique, intermittent thermodilution, has a precision of ± 20% or less. The combination of two precisions of ± 20% equates to a total error of ± 28.3%, which is commonly rounded up to ± 30%. Thus, finding a percentage error of less than ± 30% should equate to the new tested technique having an error similar to the reference, which therefore should be acceptable. In a worked example in this paper, we discuss the limitations of this approach, in particular in regard to the situation in which the reference technique may be either more or less precise than would normally be expected. This can lead to inappropriate conclusions being drawn from data acquired in validation studies of new monitoring technologies. We conclude that it is not acceptable to present comparison studies quoting percentage error as an acceptability criteria without reporting the precision of the reference technique.

The precision of PiCCO®measurements in hypothermic post‐cardiac arrest patients

“…Imprecise data may lead to erroneous judgments; if a monitoring device or technique is highly precise, it will be able to detect small changes in a variable. However, if it is imprecise, it will only be able to detect much larger changes [25]. Therefore, it is necessary to establish the number of injections that are required for a measurement to achieve a statistically relevant level of precision and least significant change.…”

“…Precision is considered to be 'good' when variability due to random error in the measurements is low [25]. The coefficient of variation was determined and then used to calculate the precision and the least significant change.…”

“…The least significant change is referred to as the minimum change measured by a device that is needed to recognise a real change and can be described by the following equation [23,25,38]:…”

“…Several studies have investigated the accuracy of these values [17,[20][21][22]], but few have described the precision of this technique [23,24]. Accuracy refers to how close a measurement is to the actual or real value, and precision refers to how close repeated measurements are to each other [25]. However, some authors have argued that hypothermia may influence the precision of the system by increasing thermal noise and thereby reducing the signal-to-noise ratio [26,27].…”

Assessment of agreement and trending between haemodynamic monitors is still challenging

“…The benchmark at ±30 % (±28.3 % to be more precise) refers to the assumption that intermittent thermodilution (ITD) by the pulmonary artery catheter has a precision of ±20 % and means that if PE lies within a range of ±30 %, methods are interchangeable with ITD as they show a comparable individual error. In this study, TEE showed a superior precision (7.2 %) than ITD and, consequently, redetermining the range of an acceptable PE could be considered [3]. Claiming the new methods to have at least the same individual error as the TEE would result in an acceptable PE of ±10.2 %.…”