Automated Extraction of Diagnostic Criteria From Electronic Health Records for Autism Spectrum Disorders: Development, Evaluation, and Application

Abstract

BackgroundElectronic health records (EHRs) bring many opportunities for information utilization. One such use is the surveillance conducted by the Centers for Disease Control and Prevention to track cases of autism spectrum disorder (ASD). This process currently comprises manual collection and review of EHRs of 4- and 8-year old children in 11 US states for the presence of ASD criteria. The work is time-consuming and expensive.ObjectiveOur objective was to automatically extract from EHRs the description of behaviors noted by the clinicians in evidence of the diagnostic criteria in the Diagnostic and Statistical Manual of Mental Disorders (DSM). Previously, we reported on the classification of entire EHRs as ASD or not. In this work, we focus on the extraction of individual expressions of the different ASD criteria in the text. We intend to facilitate large-scale surveillance efforts for ASD and support analysis of changes over time as well as enable integration with other relevant data.MethodsWe developed a natural language processing (NLP) parser to extract expressions of 12 DSM criteria using 104 patterns and 92 lexicons (1787 terms). The parser is rule-based to enable precise extraction of the entities from the text. The entities themselves are encompassed in the EHRs as very diverse expressions of the diagnostic criteria written by different people at different times (clinicians, speech pathologists, among others). Due to the sparsity of the data, a rule-based approach is best suited until larger datasets can be generated for machine learning algorithms.ResultsWe evaluated our rule-based parser and compared it with a machine learning baseline (decision tree). Using a test set of 6636 sentences (50 EHRs), we found that our parser achieved 76% precision, 43% recall (ie, sensitivity), and >99% specificity for criterion extraction. The performance was better for the rule-based approach than for the machine learning baseline (60% precision and 30% recall). For some individual criteria, precision was as high as 97% and recall 57%. Since precision was very high, we were assured that criteria were rarely assigned incorrectly, and our numbers presented a lower bound of their presence in EHRs. We then conducted a case study and parsed 4480 new EHRs covering 10 years of surveillance records from the Arizona Developmental Disabilities Surveillance Program. The social criteria (A1 criteria) showed the biggest change over the years. The communication criteria (A2 criteria) did not distinguish the ASD from the non-ASD records. Among behaviors and interests criteria (A3 criteria), 1 (A3b) was present with much greater frequency in the ASD than in the non-ASD EHRs.ConclusionsOur results demonstrate that NLP can support large-scale analysis useful for ASD surveillance and research. In the future, we intend to facilitate detailed analysis and integration of national datasets.

A Year of Papers Using Biomedical Texts: Findings from the Section on Natural Language Processing of the IMIA Yearbook

“…When addressing more standard issues, methods involved often show mature results and permit to go further in the investigation of research questions. The clinical documents, mainly in English, continue to be widely exploited when looking for various types of information, such as the identification of health outcomes 25 , adverse drug events 9 , 26 , 27 , diagnostic criteria for autism spectrum disorders 28 , sentiment analysis through the subjective expressions made by clinicians 29 , similar clinical notes 30 , or temporal segmentation in patient histories 10 . According to the tasks aimed, different approaches are exploited (machine learning and deep learning, rule-based).…”

Findings from the 2019 International Medical Informatics Association Yearbook Section on Health Information Management

“…Their gold standard comparison was ICD-9 codes and the researchers found that NLP showed potential. Leroy et al , focused on using NLP methods applied to clinical records to identify diagnostic criteria for Autism Spectrum Disorders (ASD) 19 . Although this research was done on a clinical dataset collected prior to the most recent version of the criteria for ASD 20 and used the earlier criteria, the approach showed potential to facilitate identification of criteria for research as well as ASD surveillance in the general population.…”

Automated Phenotyping Tool for Identifying Developmental Language Disorder Cases in Health Systems Data (APT-DLD): A New Research Algorithm for Deployment in Large-Scale Electronic Health Record Systems

“…Notably, for multi-etiological syndromes such as DLD, it has in the past been time consuming and complicated to conduct a systematic chart review to achieve a well-defined phenotype for cohort identification. However, recent works on autism (Bush et al, 2017;Leroy et al, 2018;Lingren et al, 2016), mental illness (Lyalina et al, 2013;McCoy et al, 2018), and even infectious diseases (Chiu & Hripcsak, 2017) demonstrate the feasibility of applying automated algorithmic procedures to EHRs for phenotyping of complex diseases and disorders. Here, we developed, evaluated, and validated a rule-based algorithm, APT-DLD, that automatically classifies cases of DLD from EHRs, to address the challenge of identifying large cohorts of DLD within health systems.…”