Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomas

Abstract

AimsTo evaluate the accuracy of Chat Generative Pre-trained Transformer (ChatGPT) powered by GPT-4 in histopathological image detection and classification of colorectal adenomas using the diagnostic consensus provided by pathologists as a reference standard.MethodsA study was conducted with 100 colorectal polyp photomicrographs, comprising an equal number of adenomas and non-adenomas, classified by two pathologists. These images were analysed by classic GPT-4 for 1 time in October 2023 and custom GPT-4 for 20 times in December 2023. GPT-4’s responses were compared against the reference standard through statistical measures to evaluate its proficiency in histopathological diagnosis, with the pathologists further assessing the model’s descriptive accuracy.ResultsGPT-4 demonstrated a median sensitivity of 74% and specificity of 36% for adenoma detection. The median accuracy of polyp classification varied, ranging from 16% for non-specific changes to 36% for tubular adenomas. Its diagnostic consistency, indicated by low kappa values ranging from 0.06 to 0.11, suggested only poor to slight agreement. All of the microscopic descriptions corresponded with their diagnoses. GPT-4 also commented about the limitations in its diagnoses (eg, slide diagnosis best done by pathologists, the inadequacy of single-image diagnostic conclusions, the need for clinical data and a higher magnification view).ConclusionsGPT-4 showed high sensitivity but low specificity in detecting adenomas and varied accuracy for polyp classification. However, its diagnostic consistency was low. This artificial intelligence tool acknowledged its diagnostic limitations, emphasising the need for a pathologist’s expertise and additional clinical context.

Large Language Models in Pathology: A Comparative Study of ChatGPT and Bard with Pathology Trainees on Multiple-Choice Questions

“…[19] Although integrating image-based questions could provide a more comprehensive assessment of LLMs in pathology, recent literature indicates that the accuracy of medical image analysis by these models remains suboptimal. [20][21][22] Thus, incorporating image-based questions might not yet yield reliable comparisons at this point and could require further technological advancements and validations. Second, the relatively small number of trainee participants is a notable limitation.…”

The First Look at Code-Free Custom GPTs in Medicine: An Ophthalmology Perspective (Preprint)

“…Although such a conversation is currently possible with standard GPT-4, GPT-4's knowledge on this topic will be limited and the surgeon will not be able to obtain detailed information. Unfortunately, the image analysis function of GPTs is currently not sufficient we underscored the possibilities through text only data, we believe that with the improvement of image recognition function, many new areas of application will arise [21].…”

“…We found two studies investigating OpenAI's system [20,21]. In the first study, an anesthesia team uploaded their department protocols to an cGPT, which achieved 90.8% accuracy on text-based information.…”

Applications of Large Language Models in Pathology

“…Off-the-shelf MLLMs, such as GPT4v, are not well-trained on pathology images, so are not very suitable for analyzing histopathological images. In one study involving 100 colorectal polyp photomicrographs, ChatGPT achieved a sensitivity of 74% and specificity of 36% in adenoma detection [87]. Sievert et al trained ChatGPT with 16 oropharyngeal confocal laser endomicroscopy images (8 with squamous cell carcinoma, 8 with normal mucosa) and it was tested with 139 images (83 with squamous cell carcinoma and 56 with healthy normal mucosa), achieving an overall accuracy of 71%, demonstrating an ability for fewshot learning.…”