Appropriate Statistics for Determining Chance-Removed Interpractitioner Agreement

Abstract

In all cases, overall agreement was much lower with FK than Gwet's AC2. Larger differences occurred when the data were more free marginal. Inter-rater agreement determined with FK statistics is unlikely to be correct unless it can be shown that the data from which agreement is determined are, in fact, fixed marginal. It follows that results obtained on agreement between practitioners with FK are probably incorrect. It is shown that inter-rater agreement evaluated with AC2 statistic is an appropriate measure when fixed marginal data are neither expected nor guaranteed. The AC2 statistic should be used as the standard statistical approach for determining agreement between practitioners.

A Novel Approach to Describing Traditional Chinese Medical Patterns: The “Traditional Chinese Medical Diagnostic Descriptor”

“…However, the study carried out with the DSOMf, a format with the same theoretical properties, yielded similar agreement outcomes when 5 practitioners examined 42 subjects. 2 The DSOMf study strengthens the validity of the result obtained with the TCMDD. The two studies when considered together provide some validation.…”

“…This is a significant result, as previous TCM diagnostic reliability research has focused on single disease states. Furthermore, in the second article 2 it was revealed that most inter-rater reliability studies utilized incorrect Fleiss' kappa statistic, whereas the chanceremoved statistic developed by Gwet and calculated with software named the Agreement Coefficient 2 (AC2) 3 was identified and demonstrated as appropriate. Unlike Gwet's AC2, Fleiss' kappa is only accurate if each category is equally represented in the data analyzed (known as fixed marginality) and severely reduces agreement if they are not.…”

Complex Adaptive Systems Theory and Inter-Rater Reliability: Proposed Answers to Challenging Questions

“…Poppelwell et al compare two probabilistic statistical approaches and propose one over the other. 11 Relying on probability distributions is not congruent with TCM. In TCM, diagnosis and treatment are learning processes.…”

Inter-Rater Agreement in Traditional Chinese Medicine: On the Potential Contribution of Popplewell's Work

“…1 In the second article they present arguments for not using the Fleiss Kappa statistic as a test of agreement in Traditional Chinese Medicine (TCM) diagnostic studies and show that the Gwet AC2 test is a better statistical test to use. 2 In the third article, they present the details of the instrument Popplewell developed to address the problem of low DA in TCM studies, the ''Traditional Chinese Medical Diagnostic Descriptor'' (TCMDD), detailing the methodology and testing of the instrument. 3 There has generally been a paucity of research on the diagnostic methods and conclusions in the practice of traditional East Asian Medicine (TEAM), 4 of which TCM is the most commonly found system.…”

“…The second article identifies a limitation of the usual statistical test for DA, the Fleiss Kappa: if there are missing variables or unselected variables, the test will not properly compute. 2 I was aware of this problem in the small pilot study I conducted in the early 1990s, where a number of results would not compute due to the lack of utilization of those less common variables, 14 this limitation made my results unpublishable in a peer-reviewed journal. If the Gwet AC2 statistic test bypasses this problem and gives a more precise analysis of DA, this is an important solution and development for studies of DA in TEAM practice systems.…”