Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination

Ji, Chen; Chokshi, Sara; Hegde, Roshini; González, Javier; Iturrate, Eduardo; Aphinyanaphongs, Yindalon; Mann, David

doi:10.2196/16848

Cited by 16 publications

(10 citation statements)

References 44 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Likewise, a few prior studies have used machine learning and other analytics to examine alert compliance with the goal of alert suppression and reduction of alert future. For example, Chen et al 40 model compliance with a vaccine reminder alert and implement their model to suppress selected alerts; their study demonstrated promising results. Other prior studies (such as Wong et al 41 ) examine factors related to decision support compliance.…”

Section: Discussionmentioning

confidence: 99%

Use of machine learning to predict clinical decision support compliance, reduce alert burden, and evaluate duplicate laboratory test ordering alerts

et al. 2021

View full text Add to dashboard Cite

Objectives While well-designed clinical decision support (CDS) alerts can improve patient care, utilization management, and population health, excessive alerting may be counterproductive, leading to clinician burden and alert fatigue. We sought to develop machine learning models to predict whether a clinician will accept the advice provided by a CDS alert. Such models could reduce alert burden by targeting CDS alerts to specific cases where they are most likely to be effective. Materials and Methods We focused on a set of laboratory test ordering alerts, deployed at 8 hospitals within the Partners Healthcare System. The alerts notified clinicians of duplicate laboratory test orders and advised discontinuation. We captured key attributes surrounding 60 399 alert firings, including clinician and patient variables, and whether the clinician complied with the alert. Using these data, we developed logistic regression models to predict alert compliance. Results We identified key factors that predicted alert compliance; for example, clinicians were less likely to comply with duplicate test alerts triggered in patients with a prior abnormal result for the test or in the context of a nonvisit-based encounter (eg, phone call). Likewise, differences in practice patterns between clinicians appeared to impact alert compliance. Our best-performing predictive model achieved an area under the receiver operating characteristic curve (AUC) of 0.82. Incorporating this model into the alerting logic could have averted more than 1900 alerts at a cost of fewer than 200 additional duplicate tests. Conclusions Deploying predictive models to target CDS alerts may substantially reduce clinician alert burden while maintaining most or all the CDS benefit.

show abstract

Section: Discussionmentioning

confidence: 99%

Use of machine learning to predict clinical decision support compliance, reduce alert burden, and evaluate duplicate laboratory test ordering alerts

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The published literature on alert fatigue in the academic sphere started as early as 2007. We collected a total of 341 published items and finally selected a total of 8 [13,15,[29][30][31][32][33][34] that fit our research criteria based on the scoping review in Figure 2. We then entered these items in Microsoft Word and Excel for deeper analysis.…”

Section: Related Work and Research Focusmentioning

confidence: 99%

“…We then entered these items in Microsoft Word and Excel for deeper analysis. We summarize and sort these literatures according to their different key foci, methods, and benefits in Multimedia Appendix 2 [13,15,[29][30][31][32][33][34].…”

Section: Related Work and Research Focusmentioning

confidence: 99%

Reducing Alert Fatigue by Sharing Low-Level Alerts With Patients and Enhancing Collaborative Decision Making Using Blockchain Technology: Scoping Review and Proposed Framework (MedAlert)

Wan¹,

Satybaldy²,

Huang³

et al. 2020

J Med Internet Res

View full text Add to dashboard Cite

Background Clinical decision support (CDS) is a tool that helps clinicians in decision making by generating clinical alerts to supplement their previous knowledge and experience. However, CDS generates a high volume of irrelevant alerts, resulting in alert fatigue among clinicians. Alert fatigue is the mental state of alerts consuming too much time and mental energy, which often results in relevant alerts being overridden unjustifiably, along with clinically irrelevant ones. Consequently, clinicians become less responsive to important alerts, which opens the door to medication errors. Objective This study aims to explore how a blockchain-based solution can reduce alert fatigue through collaborative alert sharing in the health sector, thus improving overall health care quality for both patients and clinicians. Methods We have designed a 4-step approach to answer this research question. First, we identified five potential challenges based on the published literature through a scoping review. Second, a framework is designed to reduce alert fatigue by addressing the identified challenges with different digital components. Third, an evaluation is made by comparing MedAlert with other proposed solutions. Finally, the limitations and future work are also discussed. Results Of the 341 academic papers collected, 8 were selected and analyzed. MedAlert securely distributes low-level (nonlife-threatening) clinical alerts to patients, enabling a collaborative clinical decision. Among the solutions in our framework, Hyperledger (private permissioned blockchain) and BankID (federated digital identity management) have been selected to overcome challenges such as data integrity, user identity, and privacy issues. Conclusions MedAlert can reduce alert fatigue by attracting the attention of patients and clinicians, instead of solely reducing the total number of alerts. MedAlert offers other advantages, such as ensuring a higher degree of patient privacy and faster transaction times compared with other frameworks. This framework may not be suitable for elderly patients who are not technology savvy or in-patients. Future work in validating this framework based on real health care scenarios is needed to provide the performance evaluations of MedAlert and thus gain support for the better development of this idea.

show abstract

“… 5–7 In order to initialize and sustain such tools, healthcare systems must make up-front investments in technological infrastructure and organizational structures to develop, deploy, and manage these technologies. 8 , 9 Yet there has been little description about what configuration options exist or analysis of trade-offs between different strategies. 10 An evaluation of steps taken by early-adopter health systems who have deployed AI-CDS into clinical workflow would help guide other health systems that are developing or accelerating their AI-CDS strategies, facilitate collaboration, and spread adoption of AI-CDS at the clinical bedside.…”

Section: Introductionmentioning

confidence: 99%

“…To develop and deploy AI-CDS tools, healthcare systems have to identify use cases, search for vendor solutions or build homegrown AI-CDS algorithms themselves, and coordinate the technological infrastructure to deliver model results at point of care. 9 , 11–13 Decisions are required at the organizational level concerning project governance, how diverse stakeholders are incorporated into design and implementation, and long-term maintenance and quality control. 12 , 14 Technologically, AI-CDS tools can run in real time or at set time points, use single or multiple EHR data sources for training and inference, and deliver results natively in the EHR or stand-alone applications designed for new workflows.…”

Section: Introductionmentioning

confidence: 99%

A survey of extant organizational and computational setups for deploying predictive models in health systems

Kashyap

Morse

Patel

et al. 2021

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs. Materials and Methods We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care. Results We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n = 10, 50%), IT Department led (n = 2, 10%), and AI in Healthcare (AIHC) Team (n = 8, 40%). Discussion No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS. Conclusion Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.

show abstract

Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination

Cited by 16 publications

References 44 publications

Use of machine learning to predict clinical decision support compliance, reduce alert burden, and evaluate duplicate laboratory test ordering alerts

Use of machine learning to predict clinical decision support compliance, reduce alert burden, and evaluate duplicate laboratory test ordering alerts

Reducing Alert Fatigue by Sharing Low-Level Alerts With Patients and Enhancing Collaborative Decision Making Using Blockchain Technology: Scoping Review and Proposed Framework (MedAlert)

A survey of extant organizational and computational setups for deploying predictive models in health systems

Contact Info

Product

Resources

About