Reliability of shade selection using an intraoral spectrophotometer: common mistakes in reliability analysis

Abstract

We were interested to read the paper by Witkowski S and colleagues published in the June 2012 issue of Clinical Oral Investigations. The authors' aim was to evaluate the accuracy and reproducibility of human tooth shade selection using a digital spectrophotometer. They report the mean colour difference from the mean metric for measurement precision (reliability)! Also, least square test was used to assess inter-and intra-observer reliability [1]. Why did the authors not use well-known statistical tests for reliability analysis such as intra-class correlation coefficient for quantitative variables, weighted kappa for qualitative ones [2][3][4][5] or other methods such as coefficient of variance for repeatability [2][3][4][5], considering that the mean of the variables and using paired t test, least square test or correlation coefficient (r) are among the common mistakes in reliability analysis [2-5]? Regarding reliability or agreement, it is good to know that not only avoiding common mistakes in reliability analysis but also taking into account clinical importance instead of statistical significance is a crucial issue in clinical reliability researches [2][3][4][5].The authors computed the confidence interval (CI) value 5.23 (4.66-5.86) to represent the accuracy (validity) of the measurements! As the authors point out in their conclusion, the accuracy and reproducibility of dental shade selection using the tested spectrophotometer with respect to examiner and illumination conditions reflected the reliability of this device [1]. A narrow CI has nothing to do with accuracy. We have seven well-known statistical tests for validity (accuracy) analysis, as follows: sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratio positive (LR+), likelihood ratio negative (LR−) and finally odds ratio [2][3][4][5]. Briefly, accuracy and precision (validity and reliability) are two completely different methodological issues in clinical researches, having their own statistical tests, and should not be confused with each other; otherwise, misdiagnosis, mistreatment and mismanagement of the patients will be the result of such researches.

Reproducibility of the external surface position in left‐breast DIBH radiotherapy with spirometer‐based monitoring: methodological mistake

“…(1) Such descriptive results has nothing to do with reliability analysis. (2)(3)(4)(5)(6)(7)(8) Why did the authors not used well-known tests for reliability, such as intraclass correlation coefficient (ICC) or weighted kappa? (2)(3)(4)(5)(6)(7)(8) Regarding reliability or agreement, it is good to know that ICC should be used for quantitative variables and weighted kappa (not simple kappa, because kappa has its own limitations, too) for qualitative ones.…”

Reliability of on-call radiology residents’ interpretation of 64-slice CT pulmonary angiography for the detection of pulmonary embolism: methodological error

“…As the authors point out in their results, discrepancies were noted in 25 of the 215 studies (11.6%) (1). Why did the authors report their results descriptively and not use well-known statistical tests for reliability analysis (2)(3)(4)(5)? They concluded that it is reasonable to have on-call radiology residents perform the preliminary interpretations of 64-slice CT for PE studies.…”

“…They concluded that it is reasonable to have on-call radiology residents perform the preliminary interpretations of 64-slice CT for PE studies. Such a conclusion is a mistake due to misinterpretation of the results and also the inappropriate use of statistical tests for reliability and validity analysis, and should really be avoided by researchers (2)(3)(4)(5).…”

“…Briefly, for quantitative variable intraclass correlation coefficient (ICC) agreement and for qualitative variables, weighted kappa should be used with caution because kappa also has its own limitation in reliability analysis (2)(3)(4)(5).…”

“…Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio, and negative likelihood ratio as well as odds ratio (true/false results -preferably >50) are among the tests to evaluate the validity (accuracy) of a test compared to a gold standard (2)(3)(4)(5).…”