Validation of the QAMAI tool to assess the quality of health information provided by AI

Abstract

Objective: To propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms. Study design: observational and valuative study Setting. 27 surgeons from 25 academic centers worldwide. Methods: The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool. Results: The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach's alpha=0.837). The Interclass Correlation Coefficient was 0.983 (95%CI 0.973-0.991; F(29,542)=68.3; p<0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson coefficient of 0.876 (95%CI 0.859-0.891; p<0.001) Conclusions: The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

Evaluating ChatGPT-4’s performance as a digital health advisor for otosclerosis surgery

“…Recent studies have significantly contributed to understanding the potential and limitations of AI in otolaryngology, emphasizing the need for rigorous validation of AI tools before their integration into clinical practice. For instance, the development and validation of the QAMAI tool demonstrate a systematic approach to evaluate AI-generated health information, showing robust construct validity and high internal consistency which could be instrumental in ensuring the reliability of AI platforms, including ChatGPT, within otolaryngology settings ( 23 ). Furthermore, the complexity of using AI for synthesizing clinical guidelines is highlighted by the variability in AI responses compared to expert consensus, underscoring the necessity for AI to be used with caution, particularly in complex medical fields like otolaryngology ( 24 ).…”

Can AI answer my questions? Using Artificial Intelligence to help provide information for patients with a stoma

“…The QAMAI tool stems from a methodology that has been well-validated and extensively applied for evaluating the quality of health information across various platforms, including websites [9], social networks [10], and other multimedia channels [11]. Through a panel of experts' guidelines, this tool underwent validation for its construct validity, internal consistency, and reliability using 40 LLM responses on colorectal surgery [12]. Opting for a qualitative analysis enabled the capture of subtleties and nuances in user experience and perception that frequently preclude quantitative measures.…”