Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Antaki, Fares; Touma, Samir; Milad, Daniel; El-Khoury, Jonathan; Duval, Renaud

doi:10.1101/2023.01.22.23284882

Cited by 150 publications

(189 citation statements)

References 14 publications

Supporting

Mentioning

121

Contrasting

Unclassified

Order By: Relevance

“…Its answers were verified and compared against the extant literature. These results are like those reported in other subjects, including law (Choi et al, 2023), mathematics (Frieder et al, 2023), and specialized medicine, such as ophthalmology (Antaki et al, 2023). Overall, ChatGPT's academic performance was mediocre.…”

Section: Discussionsupporting

confidence: 82%

“…However, the developers at OpenAI warn users about the model's shortcomings and explicitly state that sometimes it can come up with inadequate, or even nonsensical, answers. Therefore, the results obtained in sports science and psychology in this research, like in other academic subjects (Antaki et al, 2023;Choi et al, 2023;Frieder et al, 2023), could be anticipated. While in the course of its further development, ChatGPT will likely perform better, in agreement with Hanna (2023) it won't perform miracles as it uses existing information.…”

Section: Discussionmentioning

confidence: 68%

“…In another recent study, ChatGPT performed at or close to the passing threshold on three United States Medical Licensing exams and showed a high level of agreement in its explanations (Kung et al, 2022). Further, in an Ophthalmic Knowledge Assessment Program, ChatGPT got 55.8% and 42.7% in correctness on two simulated exams (Antaki et al, 2023).…”

Section: Introductionmentioning

confidence: 89%

See 2 more Smart Citations

ChatGPT a breakthrough in science and education: Can it fail a test?

Szabó¹

2023

Preprint

View full text Add to dashboard Cite

Released less than three months ago, ChatGPT became the center of attention of scholars around the world. This artificial intelligence (AI) language model has over 100 million subscribers worldwide and generates many discussions concerning its accuracy, advantages, and threats to science and education. Its accuracy in law, linguistics, mathematics, and medicine has already been evaluated. Most results suggest that ChatGPT could generate a passing grade in these domains. Its performance in combined subjects or social sciences has yet to be tested. The large amount of information in this general area may yield more accurate performance. Still, specific subjects in the field, with controversial research findings, can lead to significant errors, which teachers and researchers could quickly spot. In this study, ChatGPT was tested on its accuracy on exercise addiction, a subject in sports sciences and psychology associated with more than 1,000 publications. ChatGPT gave several correct answers to 20 questions but failed the test with 45%. Its performance was like in other already tested subjects. However, when prompted to write a general introductory editorial on AI’s role in sports, ChatGPT performed well. Plagiarism detectors could not identify the AI-originated text, but AI detectors did. Therefore, it can be concluded that the system does a relatively good job on general issues but needs further development in more specific areas. Students and scholars cannot rely on ChatGPT to do their job. Still, future versions could yield dilemmas of originality since the system does not provide information for its source(s) of information.

show abstract

Section: Discussionsupporting

confidence: 82%

Section: Discussionmentioning

confidence: 68%

Section: Introductionmentioning

confidence: 89%

See 1 more Smart Citation

ChatGPT a breakthrough in science and education: Can it fail a test?

Szabó¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, in ophthalmology examination, Antaki et al showed that ChatGPT currently performed at the level of an average first-year resident [ 70 ]. Such a result highlights the need to focus on questions involving the assessment of critical and problem-based thinking [ 34 ].…”

Section: Discussionmentioning

confidence: 99%

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

2023

View full text Add to dashboard Cite

ChatGPT is an artificial intelligence (AI)-based conversational large language model (LLM). The potential applications of LLMs in health care education, research, and practice could be promising if the associated valid concerns are proactively examined and addressed. The current systematic review aimed to investigate the utility of ChatGPT in health care education, research, and practice and to highlight its potential limitations. Using the PRIMSA guidelines, a systematic search was conducted to retrieve English records in PubMed/MEDLINE and Google Scholar (published research or preprints) that examined ChatGPT in the context of health care education, research, or practice. A total of 60 records were eligible for inclusion. Benefits of ChatGPT were cited in 51/60 (85.0%) records and included: (1) improved scientific writing and enhancing research equity and versatility; (2) utility in health care research (efficient analysis of datasets, code generation, literature reviews, saving time to focus on experimental design, and drug discovery and development); (3) benefits in health care practice (streamlining the workflow, cost saving, documentation, personalized medicine, and improved health literacy); and (4) benefits in health care education including improved personalized learning and the focus on critical thinking and problem-based learning. Concerns regarding ChatGPT use were stated in 58/60 (96.7%) records including ethical, copyright, transparency, and legal issues, the risk of bias, plagiarism, lack of originality, inaccurate content with risk of hallucination, limited knowledge, incorrect citations, cybersecurity issues, and risk of infodemics. The promising applications of ChatGPT can induce paradigm shifts in health care education, research, and practice. However, the embrace of this AI chatbot should be conducted with extreme caution considering its potential limitations. As it currently stands, ChatGPT does not qualify to be listed as an author in scientific articles unless the ICMJE/COPE guidelines are revised or amended. An initiative involving all stakeholders in health care education, research, and practice is urgently needed. This will help to set a code of ethics to guide the responsible use of ChatGPT among other LLMs in health care and academia.

show abstract

“…The ChatGPT also shapes cultural norms and values by feeding back into society and individuals' embodied experiences. In fact, it is today raising enormous concerns in education (Alshater, 2022) and specialized work areas (Kung et al, 2022;Guo et al, 2023;Antaki et al, 2023). In summary, ChatGPT as a language model can be seen as a product of both biology and culture, as it is informed by our understanding of natural language processing and driven by the needs and priorities of human society.…”

Section: Preprint -Please Cite the Originalmentioning

confidence: 99%

Augmented Cognition: Life as we don't know it

Hipólito¹

2023

Preprint

View full text Add to dashboard Cite

This paper proposes a framework for comprehending the integration of Artificial Intelligence (AI) as Augmented Cognition (AugCog). AugCog is viewed as an emergent bio-cultural process, reflecting AI's design, implementation, and usage. Section 1 establishes smart societies as a complex system. Section 2 defends that the development of AI is analogous to the biological process of niche construction. Section 3 defines AI as a socioculturally embodied expansion by which AI is shaped by and shapes human experience, resulting in various forms of AugCog. Section 4 highlights AugCog in mixed realities such as social media, neurotechnology, and smart environments, illustrating its emergence from multiscale and interdependent sociocultural perspectives. AugCog's perspective situates the human species in the state space of evolution, signifying our existence as a species and as embodied individuals.

show abstract

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Cited by 150 publications

References 14 publications

ChatGPT a breakthrough in science and education: Can it fail a test?

ChatGPT a breakthrough in science and education: Can it fail a test?

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

Augmented Cognition: Life as we don't know it

Contact Info

Product

Resources

About