Funnel plots of patient‐reported outcomes to evaluate health‐care quality: Basic principles, pitfalls and considerations

Abstract

A funnel plot is a graphical method to evaluate health‐care quality by comparing hospital performances on certain outcomes. So far, in nephrology, this method has been applied to clinical outcomes like mortality and complications. However, patient‐reported outcomes (PROs; eg, health‐related quality of life [HRQOL]) are becoming increasingly important and should be incorporated into this quality assessment. Using funnel plots has several advantages, including clearly visualized precision, detection of volume‐effects, discouragement of ranking hospitals and easy interpretation of results. However, without sufficient knowledge of underlying methods, it is easy to stumble into pitfalls, such as overinterpretation of standardized scores, incorrect direct comparisons of hospitals and assuming a hospital to be in‐control (ie, to perform as expected) based on underpowered comparisons. Furthermore, application of funnel plots to PROs is accompanied by additional challenges related to the multidimensional nature of PROs and difficulties with measuring PROs. Before using funnel plots for PROs, high and consistent response rates, adequate case mix correction and high‐quality PRO measures are required. In this article, we aim to provide insight into the use and interpretation of funnel plots by presenting an overview of the basic principles, pitfalls and considerations when applied to PROs, using examples from Dutch routine dialysis care.

Relationship between three reproductive factors and lung cancer: a two-sample Mendelian randomization study

“…After this, we mainly used three methods to test the sensitivity of the results of our MR analysis. We tested the heterogeneity of SNPs using IVW and MR-Egger methods, and used funnel plots as validation (23,24). We apply the MR-Egger method to evaluate whether there is horizontal pleiotropy in SNPs (25).…”

Facing the challenges of PROM implementation in Dutch dialysis care: Patients’ and professionals’ perspectives

“…The use of aggregated PROM data as a benchmark for quality of care was questioned by doctors due to the large case-mix differences between centres. Recent research in a dialysis setting indeed suggests that case-mix adjustments have to be made [ 60 ], as has also been illustrated in stroke care [ 61 ]. In addition, doctors indicate that it is difficult, if not impossible, to interpret aggregated PROM results without knowing the patient, their comorbidity, their history and their personality.…”

Use of antenatal corticosteroids among infants during gestational age at 24 to 31 weeks in 57 neonatal intensive care units of China: a cross-sectional study

“…The indirectly standardized ratios for different sites were described in the funnel plot with 95% and 99.8% Poisson limits. [ 16 , 17 ] All the analysis was performed in SAS, version 9.4 (SAS Institute, Cary, North Carolina, USA) and R 3.6.1. A two-sided P value of 0.05 was used to determine the statistical significance.…”

“…Then, the indirectly standardized ratio for each site was calculated as the observed amount of ACS usage divided by the expected number of ACS usage. The indirectly standardized ratios for different sites were described in the funnel plot with 95% and 99.8% Poisson limits [16,17] . All the analysis was performed in SAS, version 9.4 (SAS Institute, Cary, North Carolina, USA) and R 3.6.1.…”