Analysis of a Web-Based Dashboard to Support the Use of National Audit Data in Quality Improvement: Realist Evaluation

Abstract

Background Dashboards can support data-driven quality improvements in health care. They visualize data in ways intended to ease cognitive load and support data comprehension, but how they are best integrated into working practices needs further investigation. Objective This paper reports the findings of a realist evaluation of a web-based quality dashboard (QualDash) developed to support the use of national audit data in quality improvement. Methods QualDash was co-designed with data users and installed in 8 clinical services (3 pediatric intensive care units and 5 cardiology services) across 5 health care organizations (sites A-E) in England between July and December 2019. Champions were identified to support adoption. Data to evaluate QualDash were collected between July 2019 and August 2021 and consisted of 148.5 hours of observations including hospital wards and clinical governance meetings, log files that captured the extent of use of QualDash over 12 months, and a questionnaire designed to assess the dashboard’s perceived usefulness and ease of use. Guided by the principles of realist evaluation, data were analyzed to understand how, why, and in what circumstances QualDash supported the use of national audit data in quality improvement. Results The observations revealed that variation across sites in the amount and type of resources available to support data use, alongside staff interactions with QualDash, shaped its use and impact. Sites resourced with skilled audit support staff and established reporting systems (sites A and C) continued to use existing processes to report data. A number of constraints influenced use of QualDash in these sites including that some dashboard metrics were not configured in line with user expectations and staff were not fully aware how QualDash could be used to facilitate their work. In less well-resourced services, QualDash automated parts of their reporting process, streamlining the work of audit support staff (site B), and, in some cases, highlighted issues with data completeness that the service worked to address (site E). Questionnaire responses received from 23 participants indicated that QualDash was perceived as useful and easy to use despite its variable use in practice. Conclusions Web-based dashboards have the potential to support data-driven improvement, providing access to visualizations that can help users address key questions about care quality. Findings from this study point to ways in which dashboard design might be improved to optimize use and impact in different contexts; this includes using data meaningful to stakeholders in the co-design process and actively engaging staff knowledgeable about current data use and routines in the scrutiny of the dashboard metrics and functions. In addition, consideration should be given to the processes of data collection and upload that underpin the quality of the data visualized and consequently its potential to stimulate quality improvement. International Registered Report Identifier (IRRID) RR2-10.1136/bmjopen-2019-033208

Quality Dashboards in Hospital Settings

“…Open-ended user feedback also indicated the need to improve design and functionality in addition to some challenges such as data integrity. 29,33,34,36 These findings suggest dashboards need continual evaluation in collaboration with users as modifications are made to ensure dashboards are useful and easy to use. Graphs were the predominant display, and all…”

“…Findings from this review align with previous reviews, [2][3][4]6,37 suggesting quality dashboards can be effectively used for sharing data and increasing communication among stakeholders, as well as for analyzing data for decision-making and implementing change. 29,30,36 Importantly, dash-boards that support initiatives or projects 21,27 to improve patient outcomes can be integrated into QI frameworks so that data can be systematically reviewed. 8,26 Furthermore, this review adds to the existing literature by evaluating dashboard characteristics associated with quality and performance improvement outcomes.…”

“…32 One dashboard was designed for health equity. 24 Eleven studies were conducted in the United States; the remaining were from the United Kingdom, 29,36 Canada, 31 France, 32 and Australia. 27…”

“…23,27,30,33 Four studies included both. 24,25,28,31 Thirteen studies 21,[23][24][25]27,[29][30][31][32][33][34][35][36] described the use of electronic dashboards with interactive functions such as filtering, customization, and drill down. Visuals to display trends and aggregate data included graphs and numbers.…”

“…Visuals to display trends and aggregate data included graphs and numbers. Bar and column graphs were most popular, 21,[23][24][25]27,29,[31][32][33][34][35][36] followed by line graphs, 21,23,24,34-36 tabular (table) presentations, 25,[30][31][32]35 numeral data, 25,29,33,34 odometers, 21,27,29 and heat maps. 24 Line graphs were most noted for displaying trends, whereas bar or column graphs and tabular presentations were popular for displaying aggregate data.…”