The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)

Abstract

BackgroundThe aim of this project was to investigate the reliability of a new 11-item quality appraisal tool for studies of diagnostic reliability (QAREL). The tool was tested on studies reporting the reliability of any physical examination procedure. The reliability of physical examination is a challenging area to study given the complex testing procedures, the range of tests, and lack of procedural standardisation.MethodsThree reviewers used QAREL to independently rate 29 articles, comprising 30 studies, published during 2007. The articles were identified from a search of relevant databases using the following string: “Reproducibility of results (MeSH) OR reliability (t.w.) AND Physical examination (MeSH) OR physical examination (t.w.).” A total of 415 articles were retrieved and screened for inclusion. The reviewers undertook an independent trial assessment prior to data collection, followed by a general discussion about how to score each item. At no time did the reviewers discuss individual papers. Reliability was assessed for each item using multi-rater kappa (κ).ResultsMulti-rater reliability estimates ranged from κ = 0.27 to 0.92 across all items. Six items were recorded with good reliability (κ > 0.60), three with moderate reliability (κ = 0.41 - 0.60), and two with fair reliability (κ = 0.21 - 0.40). Raters found it difficult to agree about the spectrum of patients included in a study (Item 1) and the correct application and interpretation of the test (Item 10).ConclusionsIn this study, we found that QAREL was a reliable assessment tool for studies of diagnostic reliability when raters agreed upon criteria for the interpretation of each item. Nine out of 11 items had good or moderate reliability, and two items achieved fair reliability. The heterogeneity in the tests included in this study may have resulted in an underestimation of the reliability of these two items. We discuss these and other factors that could affect our results and make recommendations for the use of QAREL.

Reproducibility of Interferon Gamma (IFN-γ) Release Assays. A Systematic Review

“…We assessed study quality using a modified quality appraisal tool for studies of diagnostic reliability (QAREL), a tool that has been developed for the assessment of reproducibility studies (16) and found to be reliable (17). Key components of the quality assessment were blinding, the explanation of missing data, and reporting of erroneous results.…”

Klinik araştırmacılar için Güvenilirlik ve Uyum Çalışmalarına ait Kontrol Listesi Kılavuzu’nun [GRRAS (Guidelines for Reporting Reliability and Agreement Studies)] Türkçe uyarlaması

“…alanlarda tek bir değerlendirici tarafından alınan ölçüm sonuçlarının uyumu/güvenilirliği ya da birden fazla değerlendirici tarafından alınan ölçüm sonuçlarının uyumu/güvenilirliği araştırılmak istenebilir. Ayrıca, klinisyenler, herhangi bir hastalığın tanısının konulmasında, yaygın olarak kullanılan klasik yönteme göre düşük maliyetli ve tedaviye daha kısa bir sürede cevap veren ya da tanının hızlı ve doğru konulduğu bir yöntem bulduklarında, bu yöntemi referans (eski, klasik) yöntem ile karşılaştırarak yöntemlerden elde edilen sonuçların ne derece uyumlu olduklarını belirlemek isteyebilirler 14,15 .…”

Development of Critical Appraisal Tool for Cross- Sectional Studies (CAT-CSS)

“…The CAT-CSS is a reliable tool for evaluating the quality of cross-sectional study design. Lucas et al, 2013 [26], reported that a developed quality appraisal tool for studies of diagnostic reliability showed moderate to high reliability of its criteria. Moreover, the results of reliability of the AGREE instruments indicated that the it was reliable based on the significant agreement of appraisers who tested the instrument [27].Finally, the current results achieved the recommendation of Sanderson, Tatt, & Higgins, 2007 [28],who stated that the tools for assessing research quality should be rigorously developed, valid, reliable, and easy to use.…”