Introduction: Background: Medical education is transitioning from a time-based system to a competency-based framework. In the age of Competency-Based Medical Education, however, there is a drastically increased amount of data that needs to be interpreted. With this data, however, comes an opportunity to develop predictive analytics. Machine learning is a method of data analysis that automates analytical model building. Using algorithms that iteratively learn from data, machine learning allows computers to find hidden insights without being explicitly programmed where to look. Machine learning has been successfully used in other fields to create predictive models. Objective: This study evaluates the application of neural network as a machine learning algorithm in learning from historical data in emergency residency program and predicting future resident performance. Methods: We analyzed performance data for 16 residents (PGY1-5) who were assessed at end of each shift. Performance was graded in each of the CanMEDS Roles with scores from 1 to 7 by different attending physicians who observed residents during the shift. We transformed sequences of scores for each resident to a fixed set of features and combined all of them in one dataset. We considered scores under 6 as “At Risk Resident” and scores 6 or more as “Competent Resident”, and then we separated the dataset into training and testing sets using K-Fold cross validation and trained an artificial Neural Network in order to make decision about the future situation of residents in a specific CanMEDS Role and general performance. Results: We used 5-fold cross validation to evaluate the model, one round of cross-validation involves partitioning the whole data into complementary subsets, performing the training phase on the training set, and validating the analysis on the testing set. To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds. Results of cross validation show that accuracy of model was 72%, sensitivity was 81% and specificity was 43%. Conclusion: Machine learning algorithms such (as Neural Network) have the ability to predict future resident performance on a global level and within specific domains (i.e. CanMEDS roles). Used appropriately, such information may be a valuable for monitoring resident progress.

an adult and 3.0 mmol/L in child < age 2 ) was found in 1 case when BS was checked-overall 1/501 (0.2%); adults 1/388 (0.3%), paedatric 0/113 (0.0%). Case 1-age 70 yr, GCS 12, BS 3.8 mmol/L. Conclusion: Hypoglycemia was rarely found in patients who had a pre-hospital seizure. It did not require treatment. When it was found, hypoglycemia was unlikely to be the cause of the seizure. The results are similar to the findings from other recent, retrospective, reviews. The routine determination of blood sugars in all patients who have had a seizure prior to paramedic arrival should be reconsidered.

Background Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. Objective This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. Methods NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. Results The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). Conclusions The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Equity and game theory strategies to promote gender diversity and inclusion in an academic health science center, CJC Open (2021), doi:

BACKGROUND Residents receive a numeric performance rating (e.g., 1-7 scoring scale) along with a narrative (i.e., qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE This study evaluates the accuracy of a decision support system for raters using natural language processing (NLP) and machine learning (ML). METHODS NLP was performed retrospectively on a complete dataset of narrative comments (i.e., text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using bag-of-n-grams technique with three input types: unigram, bigrams, and trigrams. Supervised machine learning models using linear regression were trained for two outputs using the original ratings and dichotomized ratings (at risk or not). Sensitivity, specificity, and accuracy metrics are reported. RESULTS The database consisted of 7,199 unique direct observation assessments, containing both narrative comments and a 3 to 7 rating in imbalanced distribution (3-5: 726, and 6-7: 4,871 ratings). Total of 141 unique raters from five different hospitals and 45 unique residents participated over the course of five academic years. When comparing the three different input types for diagnosing if a trainee would be rated low (i.e., 1-5) or high (i.e., 6 or 7), our accuracy for trigrams was (87%), bigrams (86%), and unigrams (82%). We also found that all three input types had better prediction accuracy when using a bimodal cut (e.g., lower or higher) compared to predicting performance along the full 7-scale (50-52%). CONCLUSIONS The ML models can accurately identify underperforming residents via narrative comments provided for work-based assessments. The words generated in WBAs can be a worthy dataset to augment human decisions for educators tasked with processing large volumes of narrative assessments. CLINICALTRIAL N/A