A Natural Language Processing Model to Identify Confidential Content in Adolescent Clinical Notes

Abstract

Background: The 21st Century Cures Act mandates the immediate, electronic release of health information to patients. However, in the case of adolescents, special consideration is required to ensure that confidentiality is maintained. The detection of confidential content in clinical notes may support operational efforts to preserve adolescent confidentiality while implementing information sharing. Objective: Determine if a natural language processing (NLP) algorithm can identify confidential content in adolescent clinical progress notes. Methods: 1,200 outpatient adolescent progress notes written between 2016 and 2019 were manually annotated to identify confidential content. Labeled sentences from this corpus were featurized and used to train a two-part logistic regression model, which provides both sentence-level and note-level probability estimates that a given text contains confidential content. This model was prospectively validated on a set of 240 progress notes written in May 2022. It was subsequently deployed in a pilot intervention to augment an ongoing operational effort to identify confidential content in progress notes. Note-level probability estimates were used to triage notes for review and sentence-level probability estimates were used to highlight high-risk portions of those notes to aid the manual reviewer. Results: The prevalence of notes containing confidential content was 21% (255/1200) and 22% (53/240) in the train/test and validation cohorts. The ensemble logistic regression model achieved an AUROC of 90% and 88% in the test and validation cohorts. Its use in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over completely manual note review. Discussion: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in clinical operations augmented an ongoing operational effort to identify confidential content in adolescent progress notes. These findings suggest NLP may be used to support efforts to preserve adolescent confidentiality in the wake of the information blocking mandate.

Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes

“…3 Electronic health records lack functionality to identify and filter such information. While natural language processing algorithms have shown promise, 4 there is no widely accessible solution.…”

“…2,3 Similarly, dose dependence of these associations and presence of increased psychiatric symptoms among adolescents with infrequent use are debated. 1 With increasing rates of adolescent mental health-related problems, particularly suicide, 4,5 clarification on these issues is needed to inform screening, prevention and intervention, and policy. 6 We examined associations between common substances and psychiatric symptoms among adolescents.…”

The Role of Artificial Intelligence in Mental Health: A Review

“…Nevertheless, while it's considerable promise, NLP in the context of mental health encounters significant obstacles. Privacy and confidentiality emerge as significant considerations in the context of NLP algorithms, as they necessitate access to personal data, including clinical notes and social media activities (39). The implementation of effective data anonymization and protection measures is of paramount importance in order to safeguard the sensitive information of users.…”

Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes

“…Electronic health records lack functionality to identify and filter such information, and while natural language processing algorithms have shown promise 4 , there is no widely accessible solution.…”