Interval estimation for Cohen's kappa as a measure of agreement

Blackman, Nicole; Koval, John J.

doi:10.1002/(sici)1097-0258(20000315)19:5<723::aid-sim379>3.0.co;2-a

Cited by 167 publications

(118 citation statements)

References 31 publications

Supporting

Mentioning

116

Contrasting

Unclassified

Order By: Relevance

“…The Cohen κ was 0.81 and 0.82 for collateral connection grade, 0.73 and 0.81 for Rentrop classification, 0.75 and 0.83 for wall motion score, and 0.80 and 0.87 for LGE transmurality. 23 The limits of agreement of the LGE volume (%) were −2.1±7.4% and 2.6±6.4% by Bland-Altman analysis, respectively.…”

Section: Discussionmentioning

confidence: 99%

Frequency of Myocardial Infarction and Its Relationship to Angiographic Collateral Flow in Territories Supplied by Chronically Occluded Coronary Arteries

et al. 2013

View full text Add to dashboard Cite

Background-Despite complete interruption of antegrade coronary artery flow in the setting of a chronic total occlusion (CTO), clinical recognition of myocardial infarction is often challenging. Using cardiac MRI, we investigated the frequency and extent of myocardial infarction in patients with CTO, and assessed their relationship with regional systolic function and the extent of angiographic collateral flow. Methods and Results-We included 170 consecutive patients (median age, 62 years) with angiographically documented CTO. Regional late gadolinium enhancement and wall motion score index were assessed by cardiac MRI with the use of a 17-segment model. Angiographic collateral flow was assessed by the collateral connection grade and the Rentrop score. Evidence of previous myocardial infarction was found in 25% of patients by ECG Q waves, in 69% by regional wall motion abnormality, and in 86% of patients by late gadolinium enhancement. Increased angiographic collateral flow was associated with a lower frequency of Q waves on ECG, and a lower regional wall motion score index, late gadolinium enhancement volume (%), and degree of late gadolinium enhancement transmurality (all P<0.001), as well. Conclusions-The frequency of myocardial infarction in territories subtended

show abstract

Section: Discussionmentioning

confidence: 99%

Frequency of Myocardial Infarction and Its Relationship to Angiographic Collateral Flow in Territories Supplied by Chronically Occluded Coronary Arteries

et al. 2013

View full text Add to dashboard Cite

show abstract

“…A kappa value respectively indicates poor (#0), slight (0.01-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), and almost perfect (0.81-1.00) agreement. 12 The validity of the CVM method was represented by agreement between the gold standard and estimated staging for the initial time if intraobserver agreement was acceptable. This was also calculated using the weighted kappa statistic.…”

Section: Discussionmentioning

confidence: 99%

Validity and reliability of a method for assessment of cervical vertebral maturation

Zhao

Jiang

et al. 2012

The Angle Orthodontist

View full text Add to dashboard Cite

Objective: To evaluate the validity and reliability of the cervical vertebral maturation (CVM) method with a longitudinal sample. Materials and Methods: Eighty-six cephalograms from 18 subjects (5 males and 13 females) were selected from the longitudinal database. Total mandibular length was measured on each film; an increased rate served as the gold standard in examination of the validity of the CVM method. Eleven orthodontists, after receiving intensive training in the CVM method, evaluated all films twice. Kendall's W and the weighted kappa statistic were employed. Results: Kendall's W values were higher than 0.8 at both times, indicating strong interobserver reproducibility, but interobserver agreement was documented twice at less than 50%. A wide range of intraobserver agreement was noted (40.7%-79.1%), and substantial intraobserver reproducibility was proved by kappa values (0.53-0.86). With regard to validity, moderate agreement was reported between the gold standard and observer staging at the initial time (kappa values 0.44-0.61). However, agreement seemed to be unacceptable for clinical use, especially in cervical stage 3 (26.8%). Conclusions: Even though the validity and reliability of the CVM method proved statistically acceptable, we suggest that many other growth indicators should be taken into consideration in evaluating adolescent skeletal maturation. (Angle Orthod. 2012;82:229-234.)

show abstract

“…When the true interest is agreement between raters instead of mere association between ratings, the marginal distributions should not be too disperse. Therefore, it is very important to impose the assumption of homogeneity of marginal distributions on Cohen's j (Blackman and Koval 2000;Block and Kraemer 1989;Brennan and Prediger 1981;Zwick 1988). …”

Section: Assumptions Of Cohen's Jmentioning

confidence: 99%

“…The sampling distribution of j appears to be very non-symmetric when n is small (Blackman and Koval 2000;Block and Kraemer 1989;Koval and Blackman 1996). With a large enough n, the sampling distribution of j is approximately normal so that confidence intervals (CI) and significance tests can be easily done using standard normal distribution quantiles.…”

Section: Sampling Distribution Of Cohen's Jmentioning

confidence: 99%

Meta-analysis of Cohen’s kappa

Sun

2011

Health Serv Outcomes Res Method

165

View full text Add to dashboard Cite

Cohen's j is the most important and most widely accepted measure of interrater reliability when the outcome of interest is measured on a nominal scale. The estimates of Cohen's j usually vary from one study to another due to differences in study settings, test properties, rater characteristics and subject characteristics. This study proposes a formal statistical framework for meta-analysis of Cohen's j to describe the typical interrater reliability estimate across multiple studies, to quantify between-study variation and to evaluate the contribution of moderators to heterogeneity. To demonstrate the application of the proposed statistical framework, a meta-analysis of Cohen's j is conducted for pressure ulcer classification systems. Implications and directions for future research are discussed.Keywords Cohen's j Á Inter-rater reliability Á Meta-analysis Á GeneralizabilityIn classical test theory proposed by Spearman (1904), an observed score X is expressed as the true score T plus a random error of measurement e, i.e., X = T ? e. Reliability is defined as the squared correlation between observed scores and true scores (Lord and Novick 1968). It indicates the extent to which scores produced by a particular measurement procedure are consistent and reproducible (Thorndike 2005). Reliability is an unobserved property of scores obtained from a sample on a particular test, not an inherent property of the test (Thompson 2002;Thompson and Vacha-Hasse 2000;Vacha-Hasse 1998;Vacha-Hasse et al. 2002). Therefore, it is never appropriate to claim a test is reliable or unreliable in a research article. Instead, researchers should state the scores are reliable or unreliable. Reliability estimates usually vary from one study to another due to differences in study characteristics including study settings, test properties, and subject characteristics. A test that yields reliable scores for one group of subjects in this setting may fail to yield reliable scores for a different group of subjects in another setting. Hence, understanding the generalizability of score reliability and the factors affecting score reliability becomes an important methodological issue.

show abstract

Interval estimation for Cohen's kappa as a measure of agreement

Cited by 167 publications

References 31 publications

Frequency of Myocardial Infarction and Its Relationship to Angiographic Collateral Flow in Territories Supplied by Chronically Occluded Coronary Arteries

Frequency of Myocardial Infarction and Its Relationship to Angiographic Collateral Flow in Territories Supplied by Chronically Occluded Coronary Arteries

Validity and reliability of a method for assessment of cervical vertebral maturation

Meta-analysis of Cohen’s kappa

Contact Info

Product

Resources

About