Learning From Clinical Consensus Diagnosis in India to Facilitate Automatic Classification of Dementia: Machine Learning Study

Abstract

Background The Harmonized Diagnostic Assessment of Dementia for the Longitudinal Aging Study in India (LASI-DAD) is the first and only nationally representative study on late-life cognition and dementia in India (n=4096). LASI-DAD obtained clinical consensus diagnosis of dementia for a subsample of 2528 respondents. Objective This study develops a machine learning model that uses data from the clinical consensus diagnosis in LASI-DAD to support the classification of dementia status. Methods Clinicians were presented with the extensive data collected from LASI-DAD, including sociodemographic information and health history of respondents, results from the screening tests of cognitive status, and information obtained from informant interviews. Based on the Clinical Dementia Rating (CDR) and using an online platform, clinicians individually evaluated each case and then reached a consensus diagnosis. A 2-step procedure was implemented to train several candidate machine learning models, which were evaluated using a separate test set for predictive accuracy measurement, including the area under receiver operating curve (AUROC), accuracy, sensitivity, specificity, precision, F1 score, and kappa statistic. The ultimate model was selected based on overall agreement as measured by kappa. We further examined the overall accuracy and agreement with the final consensus diagnoses between the selected machine learning model and individual clinicians who participated in the clinical consensus diagnostic process. Finally, we applied the selected model to a subgroup of LASI-DAD participants for whom the clinical consensus diagnosis was not obtained to predict their dementia status. Results Among the 2528 individuals who received clinical consensus diagnosis, 192 (6.7% after adjusting for sampling weight) were diagnosed with dementia. All candidate machine learning models achieved outstanding discriminative ability, as indicated by AUROC >.90, and had similar accuracy and specificity (both around 0.95). The support vector machine model outperformed other models with the highest sensitivity (0.81), F1 score (0.72), and kappa (.70, indicating substantial agreement) and the second highest precision (0.65). As a result, the support vector machine was selected as the ultimate model. Further examination revealed that overall accuracy and agreement were similar between the selected model and individual clinicians. Application of the prediction model on 1568 individuals without clinical consensus diagnosis classified 127 individuals as living with dementia. After applying sampling weight, we can estimate the prevalence of dementia in the population as 7.4%. Conclusions The selected machine learning model has outstanding discriminative ability and substantial agreement with a clinical consensus diagnosis of dementia. The model can serve as a computer model of the clinical knowledge and experience encoded in the clinical consensus diagnostic process and has many potential applications, including predicting missed dementia diagnoses and serving as a clinical decision support tool or virtual rater to assist diagnosis of dementia.

Identification of Drug-Induced Liver Injury Biomarkers from Multiple Microarrays Based on Machine Learning and Bioinformatics Analysis

“…In this study, we set the random seed at 234 with the set.seed function to separate the training and testing sets, and the microarrays were randomly divided into the training and testing sets in a ratio of 8:2. In fact, we found that the segmentation of the training and testing data has different division standards in the different articles, such as 90:10 [ 48 ], 85:15 [ 49 ], 80:20, 70:30 [ 50 ], 60:40 [ 51 ], and so on. We choose 8:2 for three reasons: first, it is supported by the reported literature; second, considering that the number of samples is not large enough, we wanted to make the training data for developing the model as large as possible; third, the ratio of the number of DILI samples and control samples is about 80:20.…”

Estimating the Prevalence of Dementia in India Using a Semi-Supervised Machine Learning Approach

“…About 60% of LASI-DAD participants received clinical consensus diagnoses of their dementia status [16]. A machine learning model has been developed and validated to expand the clinical consensus diagnosis of dementia to all LASI-DAD participants [17].…”

“…The ultimate model was a stochastic gradient boosting model with an area under receiver operating characteristics (ROC) curve of 0.94, indicating that the model has an almost perfect discriminative ability according to thresholds suggested in prior research [19]. Details of the machine learning model have been described elsewhere [17]. To maximize the amount of labeled data for analysis in this paper, both the clinical consensus diagnoses and the predicted diagnoses in the LASI-DAD study were used as the labeled data to develop the semi-supervised machine learning model.…”

“…In this study, the initial predictive model was trained with a random selection of 70% labeled data from the LASI-DAD study using a stochastic gradient boosting method, the same machine learning method that has been shown in our previous study to generate accurate predictions for LASI-DAD participants [17]. The initial model was then iteratively retrained using the self-training function in the R "ssc" package [28].…”

Simulating and estimating agreement in the presence of multiple raters and covariates

“…[1][2][3] Estimators for agreement are also used in other fields, such as a performance metric in machine learning. 4,5 Agreement studies are useful when the outcome is unknown or subjective (eg, Alzheimer's disease diagnosis based on neuroimaging biomarkers or the optimal machine learning model). Multiple raters may be included in agreement studies.…”