Usability Evaluation of Dashboards: A Systematic Literature Review of Tools

Almasi, Sohrab; Bahaadinbeigy, Kambiz; Ahmadi, Hossein; Sohrabei, Solmaz; Rabiei, Reza

doi:10.1155/2023/9990933

Cited by 14 publications

(4 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper outlines the protocol to explore how different initial key messages impact the level of behavioural engagement with the CARA dashboard. User research can determine a dashboard's overall performance and ensures rigorous design and implementation, assesses usefulness, operability, ease of use, satisfaction, user interface, content, as well as system capabilities [11,34].…”

Section: Discussionmentioning

confidence: 99%

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

Garzón-Orjuela,

Vornhagen,

Blake

et al. 2024

BMC Prim. Care

View full text Add to dashboard Cite

Background CARA set out to develop a data-visualisation platform to facilitate general practitioners to develop a deeper understanding of their patient population, disease management and prescribing through dashboards. To support the continued use and sustainability of the CARA dashboards, dashboard performance and user engagement have to be optimised. User research places people at the centre of the design process and aims to evaluate the needs, behaviours and attitudes of users to inform the design, development and impact of a product. Objective To explore how different initial key messages impact the level of behavioural engagement with a CARA dashboard. Methods Participating general practices can upload their practice data for analysis and visualisation in CARA dashboards. Practices will be randomised to one of three different initial landing pages: the full dashboard or one of two key messages: a between comparison (their practice prescribing with the average of all other practices) or within comparison (with practice data of the same month the previous year) with subsequent continuation to the full dashboard. Analysis will determine which of the three landing pages encourages user interaction, as measured by the number of ‘clicks’, ‘viewings’ and ‘sessions’. Dashboard usage data will be collected through Google analytics. Discussion This study will provide evidence of behavioural engagement and its metrics during the implementation of the CARA dashboards to optimise and sustain interaction. Trial registration ISRCTN32783644 (Registration date: 02/01/2024).

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

Garzón-Orjuela,

Vornhagen,

Blake

et al. 2024

BMC Prim. Care

View full text Add to dashboard Cite

show abstract

“…In the current study, the qualitative method (think-aloud) and a questionnaire will be used in the dashboard evaluation phase. Generally, questionnaires are the most commonly used tools for usability evaluation due to the simplicity of data analysis (34). However, a combination of qualitative and quantitative approaches is suggested to appropriately measure the usability of technologies (34).…”

Section: Discussionmentioning

confidence: 99%

Design and Evaluation of a Performance Dashboard for the Faculty of Allied Medical Sciences, AJA University of Medical Sciences: A Protocol for a Mixed Methods Study

Almasi,

Mehrabi,

Ghorbani

2023

Shiraz E-Med J

Self Cite

View full text Add to dashboard Cite

Background: Faculties, as educational systems, comprise various educational groups, faculty members, researchers, students, and administrative staff. The management of data records related to the performance and activities of the faculty and its members leads to better monitoring, identification of weaknesses and strengths, and, ultimately, promotion of the faculty's performance. Dashboards are data management tools that can be used for monitoring and evaluating a faculty's performance. Objectives: This study aims to develop a protocol for the design of a faculty performance dashboard with a sequential mixed methods approach. Methods: This cross-sectional study will be conducted in the Faculty of Allied Medical Sciences, AJA University of Medical Sciences, in 2023. A mixed methods study with a sequential mixed (qualitative and then quantitative) design will be conducted in four phases. First, all the resources related to the functional dashboard are reviewed to identify its operational requirements. Second, the requirements and necessities of the software are determined by qualitative (interviews) and then quantitative (Delphi) methods. In this phase, 8 people will be interviewed during the qualitative phase, and thematic analysis will be used to analyze the data. For the quantitative step, the 2-round Delphi technique will be conducted by the purposive selection of 21 individuals. Data analysis for the quantitative step will be conducted in SPSS v. 22 by using descriptive statistics, including mode, median, mean, and percentage of agreement. Third, software coding will be performed in C# programming language in Visual Studio. Finally, 15 people among faculty members and managers will be selected by using purposive sampling to evaluate the software. In this phase, the qualitative method and then the quantitative method are used for software evaluation. In the qualitative method, the think-aloud protocol will be used to evaluate usability, and in the quantitative method, the users' satisfaction with the dashboard software will be assessed using a questionnaire. The validity and reliability of the questionnaire have been confirmed previously (Cronbach's alpha: 0.94). The data will be analyzed using descriptive statistics in SPSS v. 21. Results: The paper identifies four steps that should be followed when designing and adopting performance dashboards to support student agency and empowerment. Conclusions: The final product of this study is a dashboard for monitoring, evaluating performance, and managing resources at the faculty level.

show abstract

“…Despite the proliferation of dashboards throughout healthcare, we found major opportunities to improve the development process of dashboards. Half of dashboards (n=59/119, 50.0%) did not involve end users in the development process, and even fewer (26/118, 22.0%) included formative usability testing, which are effective strategies to improve usability and adoption [219,220]. This corroborates the findings a prior systematic review of safety dashboards, which found a minority of dashboards used formative usability testing [221], and another recent scoping review which found that even when completed, usability testing is often incomplete [219].…”

Section: Principal Findings and Comparison With Prior Workmentioning

confidence: 99%

Scoping review of dashboards in health care: why, how, and for whom they have been developed, implemented, and evaluated (Preprint)

Helminski,

Sussman,

Pfeiffer

et al. 2024

Preprint

View full text Add to dashboard Cite

BACKGROUND While dashboards are frequently used by health systems to monitor and improve performance, little is known about their characteristics and the processes used to incorporate them into practice. OBJECTIVE This scoping review aimed to explore published literature on health care dashboards to summarize why, how, and for whom they have been developed, implemented, and evaluated. METHODS MEDLINE, EMBASE, Web of Science and the Cochrane Library were searched from inception through July 2020. Studies were included if they described the development or evaluation of a health care dashboard with publication since 2015. Clinical setting, purpose (categorized as clinical, administrative, or both), and end-user were extracted for all studies. Data on dashboard development, design characteristics, implementation strategies, and evaluation were extracted only for studies published from 2018-2020. RESULTS Data was extracted for 199 dashboards described in 193 publications. The most common settings in which dashboards were used were inpatient (n=81/199, 40.7%) and outpatient clinics (n=68/199, 34.2%). Most dashboards had ≥2 stated purposes (n=142/199, 71.4%); of these, 97/199 (48.7%) were administrative, 68/199 (34.2%) were clinical, and 31/199 (15.6%) met both criteria. Most dashboards included front-line clinical staff as end-users (n=154/199, 77.4%). In the 118 dashboards published from 2018-2020, half involved end-users in the design process (n=59/118, 50.0%); fewer described formative usability testing (n=26/118, 22.0%) or use of any theory or framework to guide development, implementation, or evaluation (n=24/118, 20.3%). Dashboards tended to be web-based (n=41/118, 34.7%) or embedded directly into the electronic health record (n=17/118, 14.4%). Common implementation strategies used alongside dashboards included education (n=60/118, 50.8%), audit and feedback (n=59/118, 50.0%), and advisory boards (n=54/118, 45.8%). Evaluations of dashboards (n=84/118, 71.2%) were mostly quantitative (n=60/118, 50.8%), or mixed methods, combining dashboard or health record data with qualitative findings (n=18/118, 15.2%). CONCLUSIONS Dashboards in health care have been used in diverse settings, for multiple purposes and varied end users. The complexity of many dashboards – serving many purposes and end users at once – and the frequent expectation that clinicians will use them underscores the importance of systematic design and evaluation to ensure they are having the intended outcomes. Future work is needed to identify best practices for the development, implementation, and evaluation of health care dashboards. INTERNATIONAL REGISTERED REPORT RR2-10.2196/34894

show abstract

Usability Evaluation of Dashboards: A Systematic Literature Review of Tools

Cited by 14 publications

References 66 publications

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

Design and Evaluation of a Performance Dashboard for the Faculty of Allied Medical Sciences, AJA University of Medical Sciences: A Protocol for a Mixed Methods Study

Scoping review of dashboards in health care: why, how, and for whom they have been developed, implemented, and evaluated (Preprint)

Contact Info

Product

Resources

About