Comparing dependent kappa coefficients obtained on multilevel data

Vanbelle, Sophie

doi:10.1002/bimj.201600093

Cited by 46 publications

(41 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To account for the correlation of the clustered data (ie, multiple measurement within each patient) the method proposed by Obuchowski was applied (26). Interrater reliability was calculated with the Fleiss multirater k and compared by using the Hotelling T 2 test (27). Two-sided P values of less than .05 were considered to indicate statistically significant differences.…”

Section: Resultsmentioning

confidence: 99%

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

Muehlematter¹,

Burger²,

Becker³

et al. 2019

Radiology

View full text Add to dashboard Cite

Background Recent studies have reported the additive value of combined gallium 68 (Ga)labeled Glu-urea-Lys (Ahx)-HBED-CC ligand targeting the prostate-specific membrane antigen (PSMA) (hereafter called Ga-PSMA-11) PET/MRI for the detection and localization of primary prostate cancer compared with multiparametric MRI. Purpose To compare the diagnostic accuracy and interrater agreement of multiparametric MRI and Ga-PSMA-11 PET/MRI for the detection of extracapsular extension (ECE) and seminal vesicle infiltration (SVI) in patients with prostate cancer. Materials and Methods Retrospective analysis of 40 consecutive men who underwent multiparametric MRI and Ga-PSMA-11 PET/MRI within 6 months for suspected prostate cancer followed by radical prostatectomy between April 2016 and July 2018. Four readers blinded to clinical and histopathologic findings rated the probability of ECE and SVI at multiparametric MRI and PET/MRI by using a five-point Likert-type scale. The prostatectomy specimen served as the reference standard. Accuracy was assessed with a multireader multicase analysis and by calculating reader-average areas under the receiver operating characteristics curve (AUCs), sensitivity, and specificity for ordinal and dichotomized data in a region-specific and patient-specific approach. Interrater agreement was assessed with the Fleiss multirater. Results For multiparametric MRI versus PET/MRI in ECE detection, respectively, AUC, sensitivity, and specificity in the region-specific analysis were 0.67 and 0.75 (.07), 28% (21 of 76) and 47% (36 of 76) (.09), and 94% (529 of 564) and 90% (509 of 564) (.007). For the patient-specific analysis, AUC, sensitivity, and specificity were 0.66 and 0.73 (.19), 46% (22 of 48) and 69% (33 of 48) (.04), and 75% (84 of 112) and 67% (75 of 112) (.19), respectively. For multiparametric MRI versus PET/MRI in SVI detection, respectively, AUC, sensitivity, and specificity of the region-specific analysis were 0.66 and 0.74 (.21), 35% (seven of 20) and 50% (10 of 20) (.25), and 98% (295 of 300) and 94% (282 of 300) (< .001). For the patient-specific analysis, AUC, sensitivity, and specificity were 0.65 and 0.79 (.25), 35% (seven of 20) and 55% (11 of 20) (.20), and 98% (137 of 140) and 94% (131 of 140) (.07), respectively. Interrater reliability for multiparametric MRI versus PET/MRI did not differ for ECE (, 0.46 vs 0.40; = .24) and SVI (, 0.23 vs 0.33; = .39). Conclusion Our results suggest that gallium 68 (Ga)-labeled Glu-urea-Lys (Ahx)-HBED-CC ligand targeting the prostate-specific membrane antigen (PSMA) (Ga-PSMA-11) PET/MRI and multiparametric MRI perform similarly for local staging of prostate cancer in patients with intermediate-to-high-risk prostate cancer. The increased sensitivity of Ga-PSMA-11 PET/MRI for the detection of extracapsular disease comes at the cost of a slightly reduced specificity.

show abstract

Section: Resultsmentioning

confidence: 99%

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

Muehlematter¹,

Burger²,

Becker³

et al. 2019

Radiology

View full text Add to dashboard Cite

show abstract

“…We calculated the difference between these two coefficients and summarized it in a graph. We also calculated p values to explore for statistically significant differences using an adaption of Hotelling's T 2 test described by Vanbelle, S 14 . Since we were testing the same hypothesis many times we used Holm's correction procedure to adjust the obtained p values for multiple hypothesis testing.…”

Section: Resultsmentioning

confidence: 99%

The use of spectrograms improves the classification of wheezes and crackles in an educational setting

Aviles-Solis

Storvoll

Vanbelle

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

Chest auscultation is a widely used method in the diagnosis of lung diseases. However, the interpretation of lung sounds is a subjective task and disagreements arise. New technological developments like the use of visSual representation of sounds through spectrograms could improve the agreement when classifying lung sounds, but this is not yet known. In this study, we tested if the use of spectrograms improves the agreement when classifying wheezes and crackles. To do this, we asked twenty-three medical students at UiT the Arctic University of Norway to classify 30 lung sounds recordings for the presence of wheezes and crackles. The sample contained 15 normal recordings and 15 with wheezes or crackles. The students classified the recordings in a random order twice. First sound only, then sound with spectrograms. We calculated kappa values for the agreement between each student and the expert classification with and without display of spectrograms and tested for significant improvement between these two coefficients. We also calculated Fleiss kappa for the 23 observers with and without the spectrogram. In an individual analysis comparing each student to an expert annotated reference standard we found that 13 out of 23 students had a positive change in kappa when classifying wheezes with the help of spectrograms. When classifying crackles 16 out of 23 showed improvement when spectrograms were used. In a group analysis we observed that Fleiss kappa values were k = 0.51 and k = 0.56 (p = 0.63) for classifying wheezes without and with spectrograms. For crackles, these values were k = 0.22 and k = 0.40 (p = <0.01) in the same order. Thus, we conclude that the use of spectrograms had a positive impact on the inter-rater agreement and the agreement with experts. We observed a higher improvement in the classification of crackles compared to wheezes.

show abstract

“…We calculated the probability of agreement and multirater Conger’s kappa using the delta method for the analysis of multilevel data. 20 Conger’s kappa coefficient was chosen over Fleiss’ kappa because the observers classifying the sounds were the same for all sounds. We analysed the intragroup agreement in each of the seven groups of observers when classifying the recordings for the presence of wheezes and crackles disregarding the breathing phase.…”

Section: Methodsmentioning

confidence: 99%

“…We used the statistical software ‘R’ V.3.2.1 together with the package ‘multiagree’ for the statistical analysis of kappa statistics. 21 …”

Section: Methodsmentioning

confidence: 99%

International perception of lung sounds: a comparison of classification across some European borders

et al. 2017

Self Cite

View full text Add to dashboard Cite

IntroductionLung auscultation is helpful in the diagnosis of lung and heart diseases; however, the diagnostic value of lung sounds may be questioned due to interobserver variation. This situation may also impair clinical research in this area to generate evidence-based knowledge about the role that chest auscultation has in a modern clinical setting. The recording and visual display of lung sounds is a method that is both repeatable and feasible to use in large samples, and the aim of this study was to evaluate interobserver agreement using this method.MethodsWith a microphone in a stethoscope tube, we collected digital recordings of lung sounds from six sites on the chest surface in 20 subjects aged 40 years or older with and without lung and heart diseases. A total of 120 recordings and their spectrograms were independently classified by 28 observers from seven different countries. We employed absolute agreement and kappa coefficients to explore interobserver agreement in classifying crackles and wheezes within and between subgroups of four observers.ResultsWhen evaluating agreement on crackles (inspiratory or expiratory) in each subgroup, observers agreed on between 65% and 87% of the cases. Conger’s kappa ranged from 0.20 to 0.58 and four out of seven groups reached a kappa of ≥0.49. In the classification of wheezes, we observed a probability of agreement between 69% and 99.6% and kappa values from 0.09 to 0.97. Four out of seven groups reached a kappa ≥0.62.ConclusionsThe kappa values we observed in our study ranged widely but, when addressing its limitations, we find the method of recording and presenting lung sounds with spectrograms sufficient for both clinic and research. Standardisation of terminology across countries would improve international communication on lung auscultation findings.

show abstract

Comparing dependent kappa coefficients obtained on multilevel data

Cited by 46 publications

References 37 publications

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

The use of spectrograms improves the classification of wheezes and crackles in an educational setting

International perception of lung sounds: a comparison of classification across some European borders

Contact Info

Product

Resources

About

Comparing dependent kappa coefficients obtained on multilevel data

Cited by 46 publications

References 37 publications

Diagnostic Accuracy of Multiparametric MRI versus 68Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

Diagnostic Accuracy of Multiparametric MRI versus 68Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

The use of spectrograms improves the classification of wheezes and crackles in an educational setting

International perception of lung sounds: a comparison of classification across some European borders

Contact Info

Product

Resources

About

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer

Diagnostic Accuracy of Multiparametric MRI versus ⁶⁸Ga-PSMA-11 PET/MRI for Extracapsular Extension and Seminal Vesicle Invasion in Patients with Prostate Cancer