Background Electronic medical records ( EMR s) allow identification of disease‐specific patient populations, but varying electronic cohort definitions could result in different populations. We compared the characteristics of an electronic medical record –derived atrial fibrillation ( AF ) patient population using 5 different electronic cohort definitions. Methods and Results Adult patients with at least 1 AF billing code from January 1, 2010, to December 31, 2017, were included. Based on different electronic cohort definitions, we trained 5 different logistic regression models using a labeled training data set (n=786). Each model yielded a predicted probability; patients were classified as having AF if the probability was higher than a specified cut point. Test characteristics were calculated for each model. These models were then applied to the full cohort and resulting characteristics were compared. In the training set, the comprehensive model (including demographics, billing codes, and natural language processing results) performed best, with an area under the curve of 0.89, sensitivity of 0.90, and specificity of 0.87. Among a candidate population (n=22 000), the proportion of patients identified as having AF varied from 61% in the model using diagnosis or procedure International Classification of Diseases ( ICD ) billing codes to 83% in the model using natural language processing of clinical notes. Among identified AF patients, the proportion of patients with a CHA 2 DS 2 ‐ VAS c score ≥2 varied from 69% to 85%; oral anticoagulant treatment rates varied from 50% to 66% depending on the model. Conclusions Different electronic cohort definitions result in substantially different AF study samples. This difference threatens the quality and reproducibility of electronic medical record–based research and quality initiatives.

No abstract

Background The electronic medical record contains a wealth of information buried in free text. We created a natural language processing algorithm to identify patients with atrial fibrillation (AF) using text alone. Methods AND RESULTS We created 3 data sets from patients with at least one AF billing code from 2010 to 2017: a training set (n=886), an internal validation set from site no. 1 (n=285), and an external validation set from site no. 2 (n=276). A team of clinicians reviewed and adjudicated patients as AF present or absent, which served as the reference standard. We trained 54 algorithms to classify each patient, varying the model, number of features, number of stop words, and the method used to create the feature set. The algorithm with the highest F-score (the harmonic mean of sensitivity and positive predictive value) in the training set was applied to the validation sets. F-scores and area under the receiver operating characteristic curves were compared between site no. 1 and site no. 2 using bootstrapping. Adjudicated AF prevalence was 75.1% at site no. 1 and 86.2% at site no. 2. Among 54 algorithms, the best performing model was logistic regression, using 1000 features, 100 stop words, and term frequency-inverse document frequency method to create the feature set, with sensitivity 92.8%, specificity 93.9%, and an area under the receiver operating characteristic curve of 0.93 in the training set. The performance at site no. 1 was sensitivity 92.5%, specificity 88.7%, with an area under the receiver operating characteristic curve of 0.91. The performance at site no. 2 was sensitivity 89.5%, specificity 71.1%, with an area under the receiver operating characteristic curve of 0.80. The F-score was lower at site no. 2 compared with site no. 1 (92.5% [SD, 1.1%] versus 94.2% [SD, 1.1%]; P <0.001). Conclusions We developed a natural language processing algorithm to identify patients with AF using text alone, with >90% F-score at 2 separate sites. This approach allows better use of the clinical narrative and creates an opportunity for precise, high-throughput cohort identification.

BackgroundClinical decision support tools for atrial fibrillation (AF) should include CHA2DS2- VASc scores to guide oral anticoagulant (OAC) treatment.ObjectiveWe compared automated, electronic medical record (EMR) generated CHA2DS2- VASc scores to clinician-documented scores, and report the resulting proportions of patients in the OAC treatment group.MethodsPatients were included if they had both a clinician documented and EMR-generated CHA2DS2-VASc score on the same day. EMR scores were based on billing codes, left ventricular ejection fraction from echocardiograms, and demographics; documented scores were identified using natural language processing. Patients were deemed “re-classified” if the EMR score was ≥2 but the documented score was <2, and vice versa. For the overall cohort and subgroups (sex and age group), we compared mean scores using paired t-tests and re-classification rates using chi-squared tests.ResultsAmong 5,767 patients, the mean scores were higher using EMR compared to documented scores (4.05 [SD 2.1] versus 3.13 [SD 1.8]; p<0.01) for the full cohort, and all subgroups (p<0.01 for all comparisons). If EMR scores were used to determine OAC treatment instead of documented scores, 8.3% (n=479, p<0.01) of patients would be re-classified, with 7.2% moving into and 1.1% moving out of the treatment group. Among 2,322 women, 4.7% (n=109, p<0.01) would be re-classified, with 4.1% into and 0.7% out of the treatment group. Among 3,445 men, 10.7% (n=370, p<0.01) would be re-classified, with 9.2% into and 1.5% out of the treatment group. Among 2,060 patients <65 years old, 18.1% (n=372, p<0.01) would be re-classified, with 15.8% into and 2.3% out of the treatment group. Among 1,877 patients 65-74 years old, 5.4% (n=101, p<0.01) would be re-classified, with 4.4% into and 1.0% out of the treatment group. Among 1,830 patients ≥75 years old, <1% would move into to the treatment group and none would move out of the treatment group.ConclusionsEMR-based CHA2DS2-VASc scores were, on average, almost a full point higher than the clinician-documented scores. Using EMR scores in lieu of documented scores would result in a significant proportion of patients moving into the treatment group, with the highest re-classifications rates in men and patients <65 years old.

Background: Healthcare systems lack a validated method to identify specific patient populations for quality improvement and research. We compared the performance of 5 different methods, for identifying patients with atrial fibrillation (AF) from the electronic medical record (EMR). We also evaluated the impact of each method on apparent oral anticoagulant (OAC) treatment rates. Methods: We identified 22000 patients with at least one billing code for AF from 2010 to 2017. A training subset (n=786) was classified as having clinical AF present or absent by 2 clinicians, with a third for adjudication (reference standard). We constructed 5 different models to predict the presence of AF: regression based on a published Medicare model, regression based on a published Kaiser model, 2 regression models using additional EMR data, and a natural language processing (NLP) model. For NLP, the code scans each note for target AF-specific terms and negating modifiers (“e.g. no history of AF”). Patients with at least one positive reference for AF were classified as having AF. We calculated the total number of patients that would be included based on each model and resulting OAC treatment rates using on prescription orders. Results: The NLP method had the highest accuracy, sensitivity and NPV. The Kaiser model had the highest PPV at the expense of sensitivity. Among all 22000 patients, the Kaiser model resulted in the fewest patients (11512, 52%) and the NLP model resulted in the most patients classified as AF present (18212, 83%). OAC treatment rates varied from 50% to 62%, depending on which model was used for patient selection. Conclusions: Some of the most widely-used administrative data algorithms to identify AF patients appear to have limitations, which may be improved upon using NLP. Our findings have implications for EMR-based quality improvement and research. In future revisions, we will incorporate the CHA2DS2-VASc score and examine variations in apparent OAC treatment rates.