A novel approach for the advancement of surgical education: the collaboration of surgical education fellows (CoSEF)

Kratzke, Ian M.; Lund, Sarah; Collings, Amelia T.; Doster, Dominique L.; Clanahan, Julie M.; Williamson, Andrea J. H.; Jensen, Rachel; Thelen, Angela E.; Han, Amy; Gates, Rebecca S.; Kearse, LaDonna E.

doi:10.1007/s44186-022-00040-1

Cited by 12 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We enlisted members of the Collaboration of Surgical Education Fellows (CoSEF), a network of General Surgery residents engaged in surgical education scholarship, to recruit residents to participate in this study [35]. We gave residents at each participating institution an institution-specific username and password to access the chatbot.…”

Section: Participantsmentioning

confidence: 99%

Evaluation of a Novel Large Language Model (LLM) Powered Chatbot for Oral-Boards Scenarios

Silvestri,

Roshal,

Shah

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

IntroductionWhile previous studies have demonstrated that generative artificial intelligence (AI) can pass medical licensing exams, AI’s role as an examiner in complex, interactive assessments remains unknown. AI-powered chatbots could serve as educational tools to simulate oral examination dialogues. Here, we present initial validity evidence for an AI-powered chatbot designed for general surgery residents to prepare for the American Board of Surgery (ABS) Certifying Exam (CE).MethodsWe developed a chatbot using GPT-4 to simulate oral board scenarios. Scenarios were completed by general surgery residents from six different institutions. Two experienced surgeons evaluated the chatbot across five domains: inappropriate content, missing content, likelihood of harm, extent of harm, and hallucinations. We measured inter-rater reliability to determine evaluation consistency.ResultsSeventeen residents completed a total of 20 scenarios. Commonly tested topics included small bowel obstruction (30%), diverticulitis (20%), and breast disease (15%). Based on two independent reviewers, evaluation revealed 11 to 25% of chatbot simulations had no errors and an additional 11% to 35% contained errors of minimal clinical significance. Chatbot limitations included incorrect management advice and critical omissions of information.ConclusionsThis study demonstrates the potential of an AI-powered chatbot in enhancing surgical education through oral board simulations. Despite challenges in accuracy and safety, the chatbot offers a novel approach to medical education, underscoring the need for further refinement and standardized evaluation frameworks. Incorporating domain-specific knowledge and expert insights is crucial for improving the efficacy of AI tools in medical education.

show abstract

Section: Participantsmentioning

confidence: 99%

Evaluation of a Novel Large Language Model (LLM) Powered Chatbot for Oral-Boards Scenarios

Silvestri,

Roshal,

Shah

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to inconsistent adherence to these recommendations, the process continues to be non-standardized, opaque, and susceptible to bias [ 12 , 13 ]. We, the Collaboration of Surgical Education Fellows (CoSEF), are a group of surgical residents with experience in surgical education research [ 14 ]. Additionally, many of us are involved in aspects of applicant selection at our institutions.…”

Section: Introductionmentioning

confidence: 99%

Reimagining general surgery resident selection: Collaborative innovation through design thinking

Sathe,

L'Huillier,

Moreci

et al. 2024

Surgery Open Science

View full text Add to dashboard Cite

“…We’re a group of surgical residents formally known as the Collaboration of Surgical Education Fellows (CoSEF), a multi-institutional organization of surgical education research fellows working together to foster peer mentorship, networking, and scholarly collaboration. 2 We’ve collectively reflected on our experiences as surgical interns across the country. We recognize that as you enter this new training environment, much of what you will face is outside of your immediate control.…”

mentioning

confidence: 99%

Thriving as a Surgical Intern: Three Tips From the Collaboration of Surgical Education Fellows (CoSEF)

L’Huillier

Lund

Anand

et al. 2023

Annals of Surgery Open

Self Cite

View full text Add to dashboard Cite

show abstract

A novel approach for the advancement of surgical education: the collaboration of surgical education fellows (CoSEF)

Cited by 12 publications

References 5 publications

Evaluation of a Novel Large Language Model (LLM) Powered Chatbot for Oral-Boards Scenarios

Evaluation of a Novel Large Language Model (LLM) Powered Chatbot for Oral-Boards Scenarios

Reimagining general surgery resident selection: Collaborative innovation through design thinking

Thriving as a Surgical Intern: Three Tips From the Collaboration of Surgical Education Fellows (CoSEF)

Contact Info

Product

Resources

About