Background: The 21st Century Cures Act mandates the immediate, electronic release of health information to patients. However, in the case of adolescents, special consideration is required to ensure that confidentiality is maintained. The detection of confidential content in clinical notes may support operational efforts to preserve adolescent confidentiality while implementing information sharing. Objective: Determine if a natural language processing (NLP) algorithm can identify confidential content in adolescent clinical progress notes. Methods: 1,200 outpatient adolescent progress notes written between 2016 and 2019 were manually annotated to identify confidential content. Labeled sentences from this corpus were featurized and used to train a two-part logistic regression model, which provides both sentence-level and note-level probability estimates that a given text contains confidential content. This model was prospectively validated on a set of 240 progress notes written in May 2022. It was subsequently deployed in a pilot intervention to augment an ongoing operational effort to identify confidential content in progress notes. Note-level probability estimates were used to triage notes for review and sentence-level probability estimates were used to highlight high-risk portions of those notes to aid the manual reviewer. Results: The prevalence of notes containing confidential content was 21% (255/1200) and 22% (53/240) in the train/test and validation cohorts. The ensemble logistic regression model achieved an AUROC of 90% and 88% in the test and validation cohorts. Its use in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over completely manual note review. Discussion: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in clinical operations augmented an ongoing operational effort to identify confidential content in adolescent progress notes. These findings suggest NLP may be used to support efforts to preserve adolescent confidentiality in the wake of the information blocking mandate.

Objective The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. Methods The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a “membership model”; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. Results The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. Conclusion This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.

Algorithm-enabled patient prioritization and remote patient monitoring (RPM) have been used to improve clinical workflows at Stanford and have been associated with improved glucose time-in-range in newly diagnosed youth with type 1 diabetes (T1D). This novel algorithm-enabled care model currently integrates continuous glucose monitoring (CGM) data to prioritize patients for weekly reviews by the clinical diabetes team. The use of additional data may help clinical teams make more informed decisions around T1D management. Regular exercise and physical activity are essential to increasing cardiovascular fitness, increasing insulin sensitivity, and improving overall well-being of youth and adults with T1D. However, exercise can lead to fluctuations in glycemia during and after the activity. Future iterations of the care model will integrate physical activity metrics (e.g., heart rate and step count) and physical activity flags to help identify patients whose needs are not fully captured by CGM data. Our aim is to help healthcare professionals improve patient care with a better integration of CGM and physical activity data. We hypothesize that incorporating exercise data into the current CGM-based care model will produce specific, clinically relevant information such as identifying whether patients are meeting exercise guidelines. This work provides an overview of the essential steps of integrating exercise data into an RPM program and the most promising opportunities for the use of these data.