Improving Electronic Health Record Note Comprehension With NoteAid: Randomized Trial of Electronic Health Record Note Comprehension Interventions With Crowdsourced Workers

Abstract

BackgroundPatient portals are becoming more common, and with them, the ability of patients to access their personal electronic health records (EHRs). EHRs, in particular the free-text EHR notes, often contain medical jargon and terms that are difficult for laypersons to understand. There are many Web-based resources for learning more about particular diseases or conditions, including systems that directly link to lay definitions or educational materials for medical concepts.ObjectiveOur goal is to determine whether use of one such tool, NoteAid, leads to higher EHR note comprehension ability. We use a new EHR note comprehension assessment tool instead of patient self-reported scores.MethodsIn this work, we compare a passive, self-service educational resource (MedlinePlus) with an active resource (NoteAid) where definitions are provided to the user for medical concepts that the system identifies. We use Amazon Mechanical Turk (AMT) to recruit individuals to complete ComprehENotes, a new test of EHR note comprehension.ResultsMean scores for individuals with access to NoteAid are significantly higher than the mean baseline scores, both for raw scores (P=.008) and estimated ability (P=.02).ConclusionsIn our experiments, we show that the active intervention leads to significantly higher scores on the comprehension test as compared with a baseline group with no resources provided. In contrast, there is no significant difference between the group that was provided with the passive intervention and the baseline group. Finally, we analyze the demographics of the individuals who participated in our AMT task and show differences between groups that align with the current understanding of health literacy between populations. This is the first work to show improvements in comprehension using tools such as NoteAid as measured by an EHR note comprehension assessment tool as opposed to patient self-reported scores.

Evaluating the Effectiveness of NoteAid in a Community Hospital Setting: Randomized Trial of Electronic Health Record Note Comprehension Interventions With Patients

“…For both samples, access to NoteAid significantly improved ComprehENotes scores (RQ1). To the best of our knowledge, NoteAid is the only tool available that has been shown to improve patient health literacy, both in this work and in prior work [ 25 , 26 ]. Samples recruited from these two sources varied, in particular with regard to age and education.…”

“…Prior work [ 24 ] has shown that ComprehENotes test scores are consistent with demographic expectations with regard to health literacy (ie, less educated respondents score lower than more educated respondents). In addition, providing access to lay definitions for medical terms via NoteAid is associated with higher scores on ComprehENotes [ 26 ].…”

“…The ComprehENotes test consists of multiple-choice questions generated by groups of physicians and medical researchers, and validated using item response theory (IRT). Research has shown that providing access to NoteAid, an educational intervention tool that automatically defines medical terms in lay terms [ 18 , 25 ], improves scores on ComprehENotes items [ 26 ]. However, participants in these studies were recruited from the Amazon Mechanical Turk (AMT) platform, and thus may not represent the typical patient population [ 27 , 28 ].…”

“…As a secondary goal, we sought to analyze the differences in performance between participants at the community hospital and participants on the AMT platform. Prior work has shown that NoteAid leads to improved scores on the ComprehENotes test for AMT participants [ 26 ]. However, overall scores for all participants on AMT were relatively high, and the population of AMT participants did not include groups typically at higher risk for low health literacy.…”

Evaluation of lexical clarification by patients reading their clinical notes: a quasi-experimental interview study

“…A strong point of our study is that we read medical correspondence from personal EHRs of the patients. Earlier studies did not use the records from patients themselves [ 11 – 13 ] and have not reported the precision and recall of the functionality that was evaluated. It can be expected that they had a similarly low performance that varied among different patients and notes, and that the increase in comprehension might be lower, when these studies would have used the actual records from patients themselves.…”