Modeling physician variability to prioritize relevant medical record information

Abstract

Objective Patient information can be retrieved more efficiently in electronic medical record (EMR) systems by using machine learning models that predict which information a physician will seek in a clinical context. However, information-seeking behavior varies across EMR users. To explicitly account for this variability, we derived hierarchical models and compared their performance to non-hierarchical models in identifying relevant patient information in intensive care unit (ICU) cases. Materials and Methods Critical care physicians reviewed ICU patient cases and selected data items relevant for presenting at morning rounds. Using patient EMR data as predictors, we derived hierarchical logistic regression (HLR) and standard logistic regression (LR) models to predict their relevance. Results In 73 pairs of HLR and LR models, the HLR models achieved an area under the ROC curve of 0.81, 95% CI [0.80, 0.82], which was statistically significantly higher than that of LR models (0.75, 95% CI [0.74-0.76]). Further, the HLR models achieved statistically significantly lower expected calibration error (0.07, 95% CI [0.06-0.08]) than LR models (0.16, 95% CI [0.14-0.17]). Discussion The physician reviewers demonstrated variability in selecting relevant data. Our results show that HLR models perform significantly better than LR models with respect to both discrimination and calibration. This is likely due to explicitly modeling physician-related variability. Conclusion Hierarchical models can yield better performance when there is physician-related variability as in the case of identifying relevant information in the EMR.

A Simple Electronic Medical Record System Designed for Research

“…The initial use case for the Simple EMR System was in support of the Learning EMR project where our goal was to build a data-driven method for prioritizing patient information in the EMR [14, 15, 16]. The project developed machine learning models to highlight patient data that are predicted to be relevant in the context of a specific clinical task [17].…”

Evaluation of Eye-tracking for a Decision Support Application

“…An intriguing application of eye-tracking in EMR systems is to enable clinical decision support applications. For example, we have used eye-tracking to collect training data for machine learning models of information-seeking behavior in an EMR system to identify and highlight data in the EMR that are likely to be relevant to the user [6, 7, 8]. The availability of inexpensive eye-tracking devices makes the broad deployment of eye-tracking enabled systems feasible.…”