A New Interpretation of the Weighted Kappa Coefficients

Vanbelle, Sophie

doi:10.1007/s11336-014-9439-4

Cited by 120 publications

(92 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The time between responses was 12–14 days. We used weighted Cohen’s kappa [39] with linear and quadratic weights [40]. Additionally, a modified weight of identical answers as 1, directly adjacent as 0.8 and all others as 0 was used, since we expected the majority of retest responses to be within ± 1 of the test response.…”

Section: Methodsmentioning

confidence: 99%

Development and validation of the ExPRESS instrument for primary health care providers’ evaluation of external supervision

Schriver

Cubaka

Vedsted

et al. 2018

Global Health Action

View full text Add to dashboard Cite

Background: External supervision of primary health care facilities to monitor and improve services is common in low-income countries. Currently there are no tools to measure the quality of support in external supervision in these countries. Aim: To develop a provider-reported instrument to assess the support delivered through external supervision in Rwanda and other countries. Methods: “External supervision: Provider Evaluation of Supervisor Support” (ExPRESS) was developed in 18 steps, primarily in Rwanda. Content validity was optimised using systematic search for related instruments, interviews, translations, and relevance assessments by international supervision experts as well as local experts in Nigeria, Kenya, Uganda and Rwanda. Construct validity and reliability were examined in two separate field tests, the first using exploratory factor analysis and a test–retest design, the second for confirmatory factor analysis. Results: We included 16 items in section A (‘The most recent experience with an external supervisor’), and 13 items in section B (‘The overall experience with external supervisors’). Item-content validity index was acceptable. In field test I, test–retest had acceptable kappa values and exploratory factor analysis suggested relevant factors in sections A and B used for model hypotheses. In field test II, models were tested by confirmatory factor analysis fitting a 4-factor model for section A, and a 3-factor model for section B. Conclusions: ExPRESS is a promising tool for evaluation of the quality of support of primary health care providers in external supervision of primary health care facilities in resource-constrained settings. ExPRESS may be used as specific feedback to external supervisors to help identify and address gaps in the supervision they provide. Further studies should determine optimal interpretation of scores and the number of respondents needed per supervisor to obtain precise results, as well as test the functionality of section B.

show abstract

Section: Methodsmentioning

confidence: 99%

Development and validation of the ExPRESS instrument for primary health care providers’ evaluation of external supervision

Schriver

Cubaka

Vedsted

et al. 2018

Global Health Action

View full text Add to dashboard Cite

show abstract

“…With ordered categories, there is usually more disagreement between the classifiers on adjacent categories than on categories that are further apart. Weighted kappa allows the user to describe the closeness between categories using weights (Vanbelle 2015;Warrens 2013Warrens , 2014. The real number 0 ≤ w ij ≤ 1 denotes the weight corresponding to cell (i, j) of tables {π ij } and {π i+ π +j }.…”

Section: Notation and Weighted Kappamentioning

confidence: 99%

“…The agreement between the observers can be used to investigate the reliability of the rating scale. Standard tools for quantifying agreement between classifications with identical categories are Cohen's kappa in the case of nominal categories (Yang and Zhou 2014;Warrens 2010b), and weighted kappa in the case of ordinal categories (Vanbelle 2015;Yang and Zhou 2015;Warrens 2012Warrens , 2013Warrens , 2015. Both coefficients correct for agreement due to chance.…”

Section: Introductionmentioning

confidence: 99%

“…The standard weighted kappas for linear scales studied in, for example, Vanbelle (2015) and Warrens (2012Warrens ( , 2013Warrens ( , 2014Warrens ( , 2015, are not appropriate with circular scales since they require that the scale has endpoints, which is not the case with circular scales. A kappa coefficient for circular classifications with identical categories as a special case of weighted kappa was first presented in Gwet (2012, p. 63-64).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Kappa Coefficients for Circular Classifications

Warrens

Pratiwi

2016

J Classif

View full text Add to dashboard Cite

Circular classifications are classification scales with categories that exhibit a certain periodicity. Since linear scales have endpoints, the standard weighted kappas used for linear scales are not appropriate for analyzing agreement between two circular classifications. A family of kappa coefficients for circular classifications is defined. The kappas differ only in one parameter. It is studied how the circular kappas are related and if the values of the circular kappas depend on the number of categories. It turns out that the values of the circular kappas can be strictly ordered in precisely two ways. The orderings suggest that the circular kappas are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for circular kappas that tend to produce higher values.

show abstract

“…the proportion of items classified in the same category by the two observers) to avoid that dependency. These absolute coefficients are however not sensitive to the scales' inability in distinguishing between items in a population with low prevalence and kappa coefficients are therefore to be preferred (Rogot and Goldberg, 1966; Vach, 2005; Kraemer et al ., 2004; Vanbelle, 2016). …”

Section: Introductionmentioning

confidence: 99%

Comparing dependent kappa coefficients obtained on multilevel data

Vanbelle

2017

Biometrical J

Self Cite

View full text Add to dashboard Cite

Reliability and agreement are two notions of paramount importance in medical and behavioral sciences. They provide information about the quality of the measurements. When the scale is categorical, reliability and agreement can be quantified through different kappa coefficients. The present paper provides two simple alternatives to more advanced modeling techniques, which are not always adequate in case of a very limited number of subjects, when comparing several dependent kappa coefficients obtained on multilevel data. This situation frequently arises in medical sciences, where multilevel data are common. Dependent kappa coefficients can result from the assessment of the same individuals at various occasions or when each member of a group is compared to an expert, for example. The method is based on simple matrix calculations and is available in the R package “multiagree”. Moreover, the statistical properties of the proposed method are studied using simulations. Although this paper focuses on kappa coefficients, the method easily extends to other statistical measures.

show abstract

A New Interpretation of the Weighted Kappa Coefficients

Cited by 120 publications

References 30 publications

Development and validation of the ExPRESS instrument for primary health care providers’ evaluation of external supervision

Development and validation of the ExPRESS instrument for primary health care providers’ evaluation of external supervision

Kappa Coefficients for Circular Classifications

Comparing dependent kappa coefficients obtained on multilevel data

Contact Info

Product

Resources

About