ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Abstract

Objective To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent's perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants ChatGPT, a publicly available, free chatbot. Main outcome measures Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

“…Additionally, the utility of ChatGPT in health care education can involve tailoring education based on the needs of the student with immediate feedback [ 46 ]. Interestingly, a recent preprint by Benoit showed the promising potential of ChatGPT in rapidly crafting consistent realistic clinical vignettes of variable complexities that can be a valuable educational source with lower costs [ 67 ]. Thus, ChatGPT can be useful in health care education including enhanced communication skills given proper academic mentoring [ 42 , 57 , 67 ].…”

“…Interestingly, a recent preprint by Benoit showed the promising potential of ChatGPT in rapidly crafting consistent realistic clinical vignettes of variable complexities that can be a valuable educational source with lower costs [ 67 ]. Thus, ChatGPT can be useful in health care education including enhanced communication skills given proper academic mentoring [ 42 , 57 , 67 ]. However, the copyright issues should be taken into account regarding the ChatGPT-generated clinical vignettes, aside from the issue of inaccurate references [ 67 ].…”

“…Thus, ChatGPT can be useful in health care education including enhanced communication skills given proper academic mentoring [ 42 , 57 , 67 ]. However, the copyright issues should be taken into account regarding the ChatGPT-generated clinical vignettes, aside from the issue of inaccurate references [ 67 ]. Additionally, ChatGPT availability can be considered as a motivation in health care education based on the personalized interaction it provides, enabling powerful self-learning as well as its utility as an adjunct in group learning [ 30 , 33 , 36 , 57 , 58 ].…”

The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations

“…Thus, ChatGPT can be useful in healthcare education including enhanced communication skills given proper academic mentoring [22,45,67]. However, the copyright issue in ChatGPT-generated clinical vignettes besides the issue of inaccurate references should be taken into account [45].…”

“…Additionally, the utility of ChatGPT in healthcare education can involve tailoring education based on the student needs with immediate feedback [32]. Interestingly, a recent preprint by Benoit showed the promising potential of ChatGPT in rapidly crafting consistent realistic clinical vignettes of variable complexities which can be a valuable educational source with lower costs [45]. Thus, ChatGPT can be useful in healthcare education including enhanced communication skills given proper academic mentoring [22,45,67].…”