Identification of Prediabetes Discussions in Unstructured Clinical Documentation: Validation of a Natural Language Processing Algorithm

Abstract

Background Prediabetes affects 1 in 3 US adults. Most are not receiving evidence-based interventions, so understanding how providers discuss prediabetes with patients will inform how to improve their care. Objective This study aimed to develop a natural language processing (NLP) algorithm using machine learning techniques to identify discussions of prediabetes in narrative documentation. Methods We developed and applied a keyword search strategy to identify discussions of prediabetes in clinical documentation for patients with prediabetes. We manually reviewed matching notes to determine which represented actual prediabetes discussions. We applied 7 machine learning models against our manual annotation. Results Machine learning classifiers were able to achieve classification results that were close to human performance with up to 98% precision and recall to identify prediabetes discussions in clinical documentation. Conclusions We demonstrated that prediabetes discussions can be accurately identified using an NLP algorithm. This approach can be used to understand and identify prediabetes management practices in primary care, thereby informing interventions to improve guideline-concordant care.

Machine Learning in General Practice: Scoping Review of Administrative Task Support and Automation

“…Table 2 shows the six basic publication attributes of the complete set of 12 studies included following the completion of the identi cation and screening processes described above (15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26). The years of publication in the included studies range from 1996 to 2022 with 10 studies published between 2017 and 2022 (15,16,(18)(19)(20)(21)(22)(23)(24)26).…”

“…The variety of current general practice problems identi ed concern appointment scheduling (15-17, 20, 21), teleconsultation (18), care management (19,24), communication (22), healthcare recommender systems (23), user interaction with electronic medical records (25), and resource management through scheduling (26), with the most frequently occurring problem being appointment scheduling. Similarly, the data reportedly used in all studies differs both across all identi ed problems and within the same problem from different researchers, with sources largely consisting of proprietary data taken from a variety of domains, including actual general practice clinics (15,17,(20)(21)(22)(23)(24), published clinical guidelines (19), electronic healthcare databases (16), and teleconsultation recordings (18), that differ considerably in their features. In looking at the level of involvement of GPs across all studies, it is not always clearly stated to what extent they participate in the actual research and only two studies clearly state involvement of GPs (18, 26).…”

“…Roughly half of the studies include a "catch-all" approach in which the performances of several different machine learning methods are compared, however, few studies (21 − 13) employed modern data-driven methods of AI based on arti cial neural networks. Despite the variety of different machine learning methods, most studies addressing a supervised machine learning problem employed traditional evaluation measures such as accuracy, precision, recall, or F-score (15,18,22,23,24) with some further opting to use measures more often found in the health sciences such as speci city (11,13,16,19). Notably, only one study employed evaluations with human judgements (20), one study elected to evaluate with respect to time and cost reductions (26), and one study employed no evaluation at all (19).…”

“…Table 2 shows the six basic publication attributes of the complete set of 12 studies included following the completion of the identi cation and screening processes described above (15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26). The years of publication in the included studies range from 1996 to 2022 with 10 studies published between 2017 and 2022 (15,16,(18)(19)(20)(21)(22)(23)(24)26). Collectively, the studies cross eight countries (USA, Spain, Canada, Great Britain, Poland, Portugal, Australia, and New Zealand) with seven studies originating primarily from the USA (15,16,(19)(20)(21)(22)26).…”

One Clinician Is All You Need–Cardiac Magnetic Resonance Imaging Measurement Extraction: Deep Learning Algorithm Development

“…Existing approaches for extracting measurements from clinical text are often based on manually developed heuristics or machine learning methods that learn from labeled data but do not leverage pretrained language representations. Rule-based approaches [ 4 ], while computationally efficient, require substantial manual effort to construct and can suffer performance degradation with shifts in linguistic structure of reports [ 5 ]. Other work has used machine learning approaches such as support vector machines and long short-term memory models to extract measurements from clinical notes, but these approaches have required large quantities of expert annotations due to absence of pretraining [ 6 ].…”