The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research

Davis, Joshua; Van Bulck, Liesbet; Durieux, Brigitte N; Lindvall, Charlotta

doi:10.2196/53559

Cited by 21 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, a limitation of this study is the lack of temperature control, which can influence the results. Studies have shown that adjusting the "creativity" settings of Chat-GPT, which involves varying the temperature settings, has potential applications for clinicians [38]. Future research should incorporate temperature regulation to evaluate its effect on study outcomes.…”

Section: Discussionmentioning

confidence: 99%

Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment

Dehdab,

Brendlin,

Werner

et al. 2024

Jpn J Radiol

View full text Add to dashboard Cite

Purpose To assess the diagnostic accuracy of ChatGPT-4V in interpreting a set of four chest CT slices for each case of COVID-19, non-small cell lung cancer (NSCLC), and control cases, thereby evaluating its potential as an AI tool in radiological diagnostics. Materials and methods In this retrospective study, 60 CT scans from The Cancer Imaging Archive, covering COVID-19, NSCLC, and control cases were analyzed using ChatGPT-4V. A radiologist selected four CT slices from each scan for evaluation. ChatGPT-4V’s interpretations were compared against the gold standard diagnoses and assessed by two radiologists. Statistical analyses focused on accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), along with an examination of the impact of pathology location and lobe involvement. Results ChatGPT-4V showed an overall diagnostic accuracy of 56.76%. For NSCLC, sensitivity was 27.27% and specificity was 60.47%. In COVID-19 detection, sensitivity was 13.64% and specificity of 64.29%. For control cases, the sensitivity was 31.82%, with a specificity of 95.24%. The highest sensitivity (83.33%) was observed in cases involving all lung lobes. The chi-squared statistical analysis indicated significant differences in Sensitivity across categories and in relation to the location and lobar involvement of pathologies. Conclusion ChatGPT-4V demonstrated variable diagnostic performance in chest CT interpretation, with notable proficiency in specific scenarios. This underscores the challenges of cross-modal AI models like ChatGPT-4V in radiology, pointing toward significant areas for improvement to ensure dependability. The study emphasizes the importance of enhancing these models for broader, more reliable medical use.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment

Dehdab,

Brendlin,

Werner

et al. 2024

Jpn J Radiol

View full text Add to dashboard Cite

show abstract

“…Importantly, the stochastic nature of these tools can result in scenarios where the identical version of a tool, prompted in the same manner, still fails to yield consistent output across replications. There have been attempts to increase the replicability of outcomes (e.g., by controlling the model's settings; Davis et al, 2024), yet there is need for more research and investigation on reproducibility.…”

Section: Discussionmentioning

confidence: 99%

Integrating AI language models in qualitative research: Replicating interview data analysis with ChatGPT

Jalali,

Akhavan

2024

System Dynamics Review

View full text Add to dashboard Cite

The recent advent of artificial intelligence (AI) language tools like ChatGPT has opened up new opportunities in qualitative research. We revisited a previous project on obesity prevention interventions, where we developed a causal loop diagram through in‐depth interview data analysis. Utilizing ChatGPT in our replication process, we compared its results against our original approach. We discuss that ChatGPT contributes to improved efficiency and unbiased data processing; however, it also reveals limitations in context understanding. Our findings suggest that AI language tools currently have great potential to serve as an augmentative tool rather than a replacement for the intricate analytical tasks performed by humans. With ongoing advancements, AI technologies may soon offer more substantial support in enhancing qualitative research capabilities, an area that deserves more investigation. © 2024 The Authors. System Dynamics Review published by John Wiley & Sons Ltd on behalf of System Dynamics Society.

show abstract

“…The 'temperature' parameter balances ChatGPT's exploratory and conservative response capabilities, directly influencing the randomness of its responses. A higher temperature results in more random and creative responses, while a lower temperature leads to more deterministic outputs 21 . For each temperature setting, experiments were also performed using three different random seeds.…”

Section: Reasoningmentioning

confidence: 99%

Exploring the Potential of ChatGPT-4 for Clinical Decision Support in Cardiac Electrophysiology and Its Semi-Automatic Evaluation Metrics

Tieliwaerdi,

Abuduweili,

Saleh

et al. 2024

Preprint

View full text Add to dashboard Cite

Background/AimDespite extensive research in other medical fields, the capabilities of ChatGPT-4 in clinical decision support within cardiac electrophysiology (EP) remain largely unexplored. This study aims to enhance ChatGPT-4’s domain-specific expertise by employing the Retrieval-Augmented Generation (RAG) approach, which integrates up-to-date, evidence-based knowledge into ChatGPT-4’s foundational database. Additionally, we plan to explore the use of commonly used automatic evaluation metrics in natural language processing, such as BERT score, BLEURT, and cosine similarity, alongside human evaluation, to develop a semi-automatic framework. This aims to reduce dependency on exhaustive human evaluations, addressing the need for efficient and scalable assessment tools in medical decision-making, given the rapid adoption of ChatGPT-4 by the public.MethodWe analyzed five atrial fibrillation (Afib) cases and seven cardiac implantable electronic device (CIED) infection cases curated from PubMed case reports. We conducted a total of 120 experiments for Afib and 168 for CIED cases, testing each case across four temperature settings (0, 0.5, 1, 1.2) and three seed settings (1, 2, 3). ChatGPT-4’s performance was assessed under two modes: the Retrieval-Augmented Generation (RAG) mode and the Cold Turkey mode, which queries ChatGPT without external knowledge via RAG. For Afib cases, ChatGPT was asked to determine rate, rhythm, and anticoagulation options, and provide reasoning for each. For CIED cases, ChatGPT is asked to determine the presence of device infections. Accuracy metrics evaluated the determination component, while reasoning was assessed by human evaluation, BERTScore, BLEURT, and cosine similarity. A mixed effects analysis was used to compare the performance under both models across varying seeds and temperatures. Spearman’s rank correlation was used to explore the relationship between automatic metrics and human evaluation.ResultsIn this study, 120 experiments for Afib and 168 for CIED were conducted. There is no significant difference between the RAG mode and the Cold Turkey mode across various metrics including determination accuracy, reasoning similarity, and human evaluation scores, although RAG achieved higher cosine similarity scores in Afib cases (0.82 vs. 0.75) and better accuracy in CIED cases (0.70 vs. 0.66), though these differences were not statistically significant due to the small sample size. Our mixed effects analysis revealed no significant effects of temperature or method interactions, indicating stable performance across these variables. Moreover, while no individual evaluation metric, such as BERT score, BLEURT or cosine similarity, showed a high correlation with human evaluations. However, the ACC-Sim metric, which averages accuracy and cosine similarity, exhibits the highest correlation with human evaluation, with Spearman’s ρ at 0.86 and a P value < 0.001, indicating a significant ordinal correlation between ACC-Sim and human evaluation. This suggests its potential as a surrogate for human evaluation in similar medical scenarios.ConclusionOur study did not find a significant difference between the RAG and Cold Turkey methods in terms of ChatGPT-4’s clinical decision-making performance in Afib and CIED infection management. The ACC-Sim metric closely aligns with human evaluations in these specific medical contexts and shows promise for integration into a semi-automatic evaluation framework.

show abstract

The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research

Cited by 21 publications

References 9 publications

Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment

Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment

Integrating AI language models in qualitative research: Replicating interview data analysis with ChatGPT

Exploring the Potential of ChatGPT-4 for Clinical Decision Support in Cardiac Electrophysiology and Its Semi-Automatic Evaluation Metrics

Contact Info

Product

Resources

About