Kappa Coefficient Calculation Using Multiple Ratings Per Subject: A Special Communication

Haley, Stephen M.; Osberg, J. Scott

doi:10.1093/ptj/69.11.970

Cited by 78 publications

(36 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This calculation provides a reasonable estimate of the reliability of dichotomous pass/fail data. 1,25 As cited by several clinical research publications, ICC and Kappa values above 0.75 should be considered representative of high levels of reliability, while values between 0.4 and 0.75 are indicative of a fair-to-moderate level of reliability. ICC values below 0.4 should be considered representative of a poor level of reliability.…”

Section: Discussionmentioning

confidence: 98%

Reliability of the Ely's test for assessing rectus femoris muscle flexibility and joint range of motion

Peeler

Anderson

2008

Journal Orthopaedic Research

View full text Add to dashboard Cite

Rehabilitative protocols and orthopadic research are significantly influenced by the ability to perform reliable measures of specific physical attributes or functions. The hypothesis was that the Ely's test for evaluating rectus femoris flexibility and joint range of motion (ROM) is a reliable clinical tool. Participants (n ¼ 54) were between the ages of 18 and 45, and had no history of trauma. Three clinicians with orthopedic expertise assessed quadriceps flexibility and joint ROM using pass/fail and goniometer scoring systems. A retest session was completed 7 to 10 days later. Statistically, Kappa values for pass/fail scoring (intrarater X ¼ 0.52, interrater X ¼ 0.46) and ICC values (intrarater X ¼ 0.69, interrater X ¼ 0.66) for goniometer data both indicated that the Ely's test demonstrated only moderate levels of intra-and interrater reliability. Measurement error values (SEM ¼ 48, ME ¼ 48, and CV ¼ 3%) and Bland and Altman plots (with 95% Limits of Agreement) further demonstrated the degree of intrarater variance for each examiner when executing the Ely's test in a clinical setting. Results call into question the statistical reliability of the Ely's test, and provide clinicians with important information regarding the reliability limits of the Ely's test when used to clinically evaluate flexibility and joint ROM in a physically active population. More research is required to determine the variables that may confound statistical reliability of this orthopedic technique that is commonly used in a clinical setting to assess function about the thigh region. ß

show abstract

Section: Discussionmentioning

confidence: 98%

Reliability of the Ely's test for assessing rectus femoris muscle flexibility and joint range of motion

Peeler

Anderson

2008

Journal Orthopaedic Research

View full text Add to dashboard Cite

show abstract

“…Inter-observer agreement and agreement with the reference agreement were estimated using different measures of agreement [27] , simple proportions of agreement (Pa) and proportions of specific agreement, and quadratic weighted Cohen's kappa coefficient (Kc) (estimated by intra-class correlation coefficient) [28][29][30][31][32][33] . The confidence intervals for proportions of agreements were estimated with binomial distribution [33] .…”

Section: Discussionmentioning

confidence: 99%

Web-based system for training and dissemination of a magnification chromoendoscopy classification

Dinis-Ribeiro¹,

Cruz‐Correia²,

Costa-Santos³

et al. 2008

WJG

View full text Add to dashboard Cite

AIM:To evaluate the use of web-based technologies to assess the learning curve and reassess reproducibility of a simplified version of a classification for gastric magnification chromoendoscopy (MC). METHODS: As part of a multicenter trial, a hybrid approach was taken using a CD-ROM, with 20 films of MC lasting 5 s each and an "autorun" file triggering a local HTML frameset referenced to a remote questionnaire through an Internet connection. Three endoscopists were asked to prospectively and independently classify 10 of these films randomly selected with at least 3 d apart. The answers were centrally stored and returned to participants together with adequate feedback with the right answer. RESULTS: For classification in 3 groups, both intra-[Cohen's kappa (κ) = 0.79-1.00 to 0.89-1.00] and inter-observer agreement increased from 1st (moderate) to 6th observation (κ = 0.94). Also, agreement with reference increased in the last observations (0.90, 1.00 and 1.00, for observers A, B and C, respectively). Validity of 100% was obtained by all observers at their 4th observation. When a 4th (sub)group was considered, inter-observer agreement was almost perfect (κ = 0.92) at 6th observation. The relation with reference clearly improved into κ (0.93-1.00) and sensitivity (75%-100%) at their 6th observations. CONCLUSION: This MC classification seems to be easily explainable and learnable as shown by excellent intra-and inter-observer agreement, and improved agreement with reference. A web system such as the one used in this study may be useful for endoscopic or other image based diagnostic procedures with respect to definition, education and dissemination.

show abstract

“…Almutairi [28] stated that the results of the accuracy analysis can be summarized by an overall accuracy percentage and a kappa statistic. In this study, the overall accuracy and kappa coefficient [29,30] were introduced to the study. Here, overall accuracy is defined to be the percentage of pixels correctly detected.…”

Section: Accuracy Assessment Methodsmentioning

confidence: 99%

“…Here, overall accuracy is defined to be the percentage of pixels correctly detected. Let p a be the proportion of agreement, p e be the chance agreement, and the kappa coefficient is defined as the proportion of agreement among raters after chance agreement has been removed [30], which can be expressed as: …”

Section: Accuracy Assessment Methodsmentioning

confidence: 99%

A Reliability-Based Multi-Algorithm Fusion Technique in Detecting Changes in Land Cover

et al. 2013

View full text Add to dashboard Cite

Detecting land use or land cover changes is a challenging problem in analyzing images. Change-detection plays a fundamental role in most of land use or cover monitoring systems using remote-sensing techniques. The reliability of individual automatic change-detection algorithms is currently below operating requirements when considering the intrinsic uncertainty of a change-detection algorithm and the complexity of detecting changes in remote-sensing images. In particular, most of these algorithms are only suited for a specific image data source, study area and research purpose. Only a number of comprehensive change-detection methods that consider the reliability of the algorithm in different implementation situations have been reported. This study attempts to explore the advantages of combining several typical change-detection algorithms. This combination is specifically designed for a highly reliable change-detection task. Specifically, a fusion approach based on reliability is proposed for an exclusive land use or land cover change-detection. First, the reliability of each candidate algorithm is evaluated. Then, a fuzzy comprehensive evaluation is used to generate a reliable change-detection approach. This evaluation is a transformation between a one-way evaluation matrix and a weight vector computed using the reliability of each candidate algorithm. Experimental results reveal that the advantages of combining these distinct change-detection techniques are evident.

show abstract

Kappa Coefficient Calculation Using Multiple Ratings Per Subject: A Special Communication

Cited by 78 publications

References 16 publications

Reliability of the Ely's test for assessing rectus femoris muscle flexibility and joint range of motion

Reliability of the Ely's test for assessing rectus femoris muscle flexibility and joint range of motion

Web-based system for training and dissemination of a magnification chromoendoscopy classification

A Reliability-Based Multi-Algorithm Fusion Technique in Detecting Changes in Land Cover

Contact Info

Product

Resources

About