Web-Based Skin Cancer Assessment and Classification Using Machine Learning and Mobile Computerized Adaptive Testing in a Rasch Model: Development Study

Wang³

et al. 2023

Self Cite

Background: The application of large language models in clinical decision support (CDS) is an area that warrants further investigation. ChatGPT, a prominent large language models developed by OpenAI, has shown promising performance across various domains. However, there is limited research evaluating its use specifically in pediatric clinical decision-making. This study aimed to assess ChatGPT’s potential as a CDS tool in pediatrics by evCDSaluating its performance on 8 common clinical symptom prompts. Study objectives were to answer the 2 research questions: the ChatGPT’s overall grade in a range from A (high) to E (low) compared to a normal sample and the difference in assessment of ChatGPT between 2 pediatricians. Methods: We compared ChatGPT’s responses to 8 items related to clinical symptoms commonly encountered by pediatricians. Two pediatricians independently assessed the answers provided by ChatGPT in an open-ended format. The scoring system ranged from 0 to 100, which was then transformed into 5 ordinal categories. We simulated 300 virtual students with a normal distribution to provide scores on items based on Rasch rating scale model and their difficulties in a range between −2 to 2.5 logits. Two visual presentations (Wright map and KIDMAP) were generated to answer the 2 research questions outlined in the objectives of the study. Results: The 2 pediatricians’ assessments indicated that ChatGPT’s overall performance corresponded to a grade of C in a range from A to E, with average scores of −0.89 logits and 0.90 logits (=log odds), respectively. The assessments revealed a significant difference in performance between the 2 pediatricians (P < .05), with scores of −0.89 (SE = 0.37) and 0.90 (SE = 0.41) in log odds units (logits in Rasch analysis). Conclusion: This study demonstrates the feasibility of utilizing ChatGPT as a CDS tool for patients presenting with common pediatric symptoms. The findings suggest that ChatGPT has the potential to enhance clinical workflow and aid in responsible clinical decision-making. Further exploration and refinement of ChatGPT’s capabilities in pediatric care can potentially contribute to improved healthcare outcomes and patient management.

Section: Methodsmentioning

confidence: 99%

Assessing ChatGPT’s capacity for clinical decision support in pediatrics: A comparative study with pediatricians using KIDMAP of Rasch analysis

Kao

Wang³

et al. 2023

Self Cite

“…If (e(θn−δi)) is replaced with exp∑k=0x(θn−(δi+τk)), the variance for person n on item i adaptive to the RSM equals the result in Equation (6). [60] Through the Newton–Raphson iteration method [61] and the person estimate and SE(trueθ^) in Equations (1) and (5), RaschOnline [59] was programed and developed.…”

Section: Methodsmentioning

confidence: 99%

“…If e (θn−δ i ) is replaced with exp x k=0 (θ n − (δ i + τ k )), the variance for person n on item i adaptive to the RSM equals the result in Equation ( 6). [60] Through the Newton-Raphson iteration method [61] and the person estimate and SE( θ) in Equations ( 1) and ( 5), RaschOnline [59] was programed and developed. Four visualizations in RaschOnline [59] were applied to present item features and person responses, including Wright Map [62] with groups, differential item functioning (DIF) [63] using forest plots, [64] item characteristic curves (ICCs), [65,66] and KIDMAP.…”

Section: Rasch Analysis Of Item Features and Person Responses Using R...mentioning

confidence: 99%

Detecting dengue fever in children using online Rasch analysis to develop algorithms for parents: An APP development and usability study

Chow

et al. 2023

Self Cite

Background: Dengue fever (DF) is a significant public health concern in Asia. However, detecting the disease using traditional dichotomous criteria (i.e., absent vs present) can be extremely difficult. Convolutional neural networks (CNNs) and artificial neural networks (ANNs), due to their use of a large number of parameters for modeling, have shown the potential to improve prediction accuracy (ACC). To date, there has been no research conducted to understand item features and responses using online Rasch analysis. To verify the hypothesis that a combination of CNN, ANN, K-nearest-neighbor algorithm (KNN), and logistic regression (LR) can improve the ACC of DF prediction for children, further research is required. Methods: We extracted 19 feature variables related to DF symptoms from 177 pediatric patients, of whom 69 were diagnosed with DF. Using the RaschOnline technique for Rasch analysis, we examined 11 variables for their statistical significance in predicting the risk of DF. Based on 2 sets of data, 1 for training (80%) and the other for testing (20%), we calculated the prediction ACC by comparing the areas under the receiver operating characteristic curve (AUCs) between DF + and DF− in both sets. In the training set, we compared 2 scenarios: the combined scheme and individual algorithms. Results: Our findings indicate that visual displays of DF data are easily interpreted using Rasch analysis; the k-nearest neighbors algorithm has a lower AUC (<0.50); LR has a relatively higher AUC (0.70); all 3 algorithms have an almost equal AUC (=0.68), which is smaller than the individual algorithms of Naive Bayes, LR in raw data, and Naive Bayes in normalized data; and we developed an app to assist parents in detecting DF in children during the dengue season. Conclusion: The development of an LR-based APP for the detection of DF in children has been completed. To help patients, family members, and clinicians differentiate DF from other febrile illnesses at an early stage, an 11-item model is proposed for developing the APP.

“…The 2 popular ML models (i.e., CNN and ANN) have also been demonstrated separately in MS Excel. [25][26][27][28][29][30][31][32] Many researchers are familiar with the use of Microsoft Excel. Until now, no research has been conducted comparing model accuracy between the 2 ML models under the MS Excel environment, particularly when the modules and Equations are detailed in Supplemental Digital Contents, as we did in this research.…”

Section: Additional Informationmentioning

confidence: 99%

“…Microsoft Excel is a familiar program to many researchers. We are motivated to compare the accuracy of 2 popular ML models (i.e., convolutional neural networks, [CNN] and artificial neural networks [ANN]) that have been demonstrated in the literature, [25][26][27][28][29][30][31][32] but with binary classes only. In general, CNN is considered to be a more powerful and accurate method of solving classification problems.…”

Section: Ml-based App For Predicting DCmentioning

confidence: 99%

An app for predicting patient dementia classes using convolutional neural networks (CNN) and artificial neural networks (ANN): Comparison of prediction accuracy in Microsoft Excel

Lin

et al. 2023

Self Cite

Background: Dementia is a progressive disease that worsens over time as cognitive abilities deteriorate. Effective preventive interventions require early detection. However, there are no reports in the literature concerning apps that have been developed and designed to predict patient dementia classes (DCs). This study aimed to develop an app that could predict DC automatically and accurately for patients responding to the clinical dementia rating (CDR) instrument. Methods: A CDR was applied to 366 outpatients in a hospital in Taiwan, with assessments on 25 and 49 items endorsed by patients and family members, respectively. The 2 models of convolutional neural networks (CNN) and artificial neural networks (ANN) were applied to examine the prediction accuracy based on 5 classes (i.e., no cognitive decline, very mild, mild, moderate, and severe) in 4 scenarios, consisting of 74 (items) in total, 25 in patients, 49 in family, and a combination strategy to select the best in the aforementioned scenarios using the forest plot. Using CDR scores in patients and their families on both axes, patients were dispersed on a radar plot. An app was developed to predict patient DC. Results: We found that ANN had higher accuracy rates than CNN with a ratio of 3:1 in the 4 scenarios. The highest accuracy rate (=93.72%) was shown in the combination scenario of ANN. A significant difference was observed between the CNN and ANN in terms of the accuracy rate. An available ANN-based app for predicting DC in patients was successfully developed and demonstrated in this study. Conclusion: On the basis of a combination strategy and a decision rule, a 74-item ANN model with 285 estimated parameters was developed and included. The development of an app that will assist clinicians in predicting DC in clinical settings is required in the near future.