Participatory design of probability-based decision support tools for in-hospital nurses

Jeffery, Alvin D.; Novak, Laurie L.; Kennedy, Betsy; Dietrich, Mary S.; Mion, Lorraine C.

doi:10.1093/jamia/ocx060

Cited by 36 publications

(40 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, access to raw patient data (e.g., laboratory values, vital signs, interventions) was seen as useful for assessing the clinical credibility and utility of predictions and explanations. This finding is consistent with results from studies by Wang et al [15] and Jeffery et al, [42] both of whom also found that providers utilized raw patient data when working with probability-based decision support systems to verify predictions and to integrate them with their clinical knowledge.…”

Section: Discussionsupporting

confidence: 90%

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Barda

Horvat

Hochheiser

2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework for the design of user-centered displays of explanations. This new framework served as the basis for qualitative inquiries and design review sessions with critical care nurses and physicians that informed the design of a user-centered explanation display for an ML-based prediction tool. Methods We used our framework to propose explanation displays for predictions from a pediatric intensive care unit (PICU) in-hospital mortality risk model. Proposed displays were based on a model-agnostic, instance-level explanation approach based on feature influence, as determined by Shapley values. Focus group sessions solicited critical care provider feedback on the proposed displays, which were then revised accordingly. Results The proposed displays were perceived as useful tools in assessing model predictions. However, specific explanation goals and information needs varied by clinical role and level of predictive modeling knowledge. Providers preferred explanation displays that required less information processing effort and could support the information needs of a variety of users. Providing supporting information to assist in interpretation was seen as critical for fostering provider understanding and acceptance of the predictions and explanations. The user-centered explanation display for the PICU in-hospital mortality risk model incorporated elements from the initial displays along with enhancements suggested by providers. Conclusions We proposed a framework for the design of user-centered displays of explanations for ML models. We used the proposed framework to motivate the design of a user-centered display of an explanation for predictions from a PICU in-hospital mortality risk model. Positive feedback from focus group participants provides preliminary support for the use of model-agnostic, instance-level explanations of feature influence as an approach to understand ML model predictions in healthcare and advances the discussion on how to effectively communicate ML model information to healthcare providers.

show abstract

Section: Discussionsupporting

confidence: 90%

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Barda

Horvat

Hochheiser

2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

show abstract

“…In addition, any restrictions in the coding space will necessitate, at least for the purposes of direct care, efficient systems to capture annotations (unstructured notes). This importance of narrative description for enabling more nuanced decision making is highlighted across several included studies; the ability to 'paint a picture' over time [56] of a patient's care trajectory is an important dimension that is currently not consistently captured in contemporary EHR systems. Agreement on the standards to be used, and the data to be collected at the point of care, is a precondition for ensuring effective decision support, and the reuse of data for quality of care indicators.…”

Section: Discussionmentioning

confidence: 99%

“…Agreement on the standards to be used, and the data to be collected at the point of care, is a precondition for ensuring effective decision support, and the reuse of data for quality of care indicators. Further research to focus on the types of cognitive support health professionals require [57], including the use of participatory design approaches [56] will help to ensure systems are fully integrated with nurse workflow. Identifying what decisions need to be supported and for whom (based on user characteristics such as expertise), may also ensure better tailoring of CDS interventions (moving away from a proliferation of alerts) and a tighter focus on safety and quality-critical issues (such as prescribing).…”

Section: Discussionmentioning

confidence: 99%

“…Currently there is considerable variability in how EHR systems are built, with potential consequences for patient safety [63]. Designing systems that can provide tailored feedback to clinicians, using data displays and visualization techniques [56] have the potential to both provide an incentive for accurate and timely data documentation, and provide the type of cognitive support clinicians need to make patient care decisions.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Reinterpreting the nursing record for an electronic context

Hardiker

Dowding

Dykes

et al. 2019

International Journal of Medical Informatics

View full text Add to dashboard Cite

Background: This article seeks to facilitate the re-imagining of nursing records purposefully within an electronic context. It questions existing approaches to nursing documentation, critically examines existing nursing record systems and identifies new requirements. Methods: A comprehensive literature review was conducted to identify themes, that might meaningfully contribute to a new approach to nursing record systems development, around four key interrelated areas-standards, decision making, abstraction and summarization, and documenting. Studies were analyzed using narrative synthesis to provide a critical analysis of the current 'state of the art', and recommendations for the future. Results: Included studies collectively described aspects of current best practice, both in terms of nursing record systems themselves, and how nurses and other health professionals contribute to and engage with those systems. A number of cross-cutting themes identified more novel approaches taken by nurses to systems development: going back to basics in determining purpose; firming up informatics foundations; nuancing or tailoring to suit different requirements; and engagement, involvement and participation. Conclusion: There is a paucity of research that specifically focuses on the nature of the electronic nursing record and its impact on patient care processes and outcomes. In addition to further research in these areas, there is a need: to reinterpret nurses as knowledge workers rather than as 'data collectors'; to agree on the application in practice of appropriate standards and terminologies; and to work together with system developers to change the ways in which data are captured and care is documented.

show abstract

“…Please entitle each table with case/subject number Figure 1 is a screen shot of the dashboard prototype, we have revised this and hope it is and make sure the numbers mentioned in Results are consistent with the ones in Tables. For example, in Satisfying, dashboard information (mean 7.8; SD 1.1) is not consistent with the result in Table 2 (6), and express a need for support in synthesizing the available information to paint a 'picture of the patient' over time (7). In an effort to help clinicians integrate data regarding diagnoses and treatment interventions for patients across time, a number of studies have developed techniques for visual display of patient data both at the individual and group level (8)(9)(10)(11).…”

mentioning

confidence: 89%

Usability Evaluation of a Dashboard for Home Care Nurses

2019

CIN: Computers, Informatics, Nursing

View full text Add to dashboard Cite

The introduction of electronic health records has produced many challenges for clinicians. These include integrating technology into clinical workflow and fragmentation of relevant information across systems. Dashboards, that use visualized data to summarize key patient information, have the potential to address these issues. In this paper we outline a usability evaluation of a dashboard designed for home care nurses. An iterative design process was used which consisted of a) contextual inquiry (observation and interviews) with 2 home care nurses b) rapid feedback on paper prototypes of the dashboard (10 nurses) and c) usability evaluation of the final dashboard prototype (20 nurses). Usability methods and assessments included observation of nurses interacting with the dashboard, the system usability scale (SUS) and the Questionnaire for User Interaction Satisfaction (QUIS) short form. The dashboard prototype was deemed to have high usability (mean SUS 73.2; SD 18.8) and was positively evaluated by nurse users. It is important to ensure that technology solutions such as the one proposed here, are designed with clinical users in mind, to meet their information needs. The design elements of the dashboard outlined here, could be translated to other EHRs used in home care settings.

show abstract

Participatory design of probability-based decision support tools for in-hospital nurses

Cited by 36 publications

References 40 publications

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Reinterpreting the nursing record for an electronic context

Usability Evaluation of a Dashboard for Home Care Nurses

Contact Info

Product

Resources

About