An Integrated Multi-modal Learning Method for Early-stage Knee Osteoarthritis Disease Classification

Abstract

Background: Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. Methods: In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. Results: MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Based on MMLM, the accuracy of SVM and DT are 83.45% and 81.27%, respectively, which are much higher than other feature fusion method. In addition, the visual analysis of the important features in the multi-modal data verified the relations among the different modalities. Conclusion: MMLM uses the internal correlations among different modalities to enhance the efficiency of feature extraction and fusion, which improves grade classification of early-stage knee OA.

Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting From Multimodal Data

“…First, we have eliminated the assumption of independence between non-imaging data m 0 and diagnostic predictions y 0 used in CLIMATv1 [22] since it does not hold not only in OA and AD, but also in other diseases. Specifically, Liu et al [34] provided empirical evidence of the benefit of the inclusion of non-imaging data in the knee OA grading task. The study conducted by Bird et al [65] indicated a link between human genes and AD while Li et al [66] showed that a blood test can detect the existence of amyloid-beta plaques in the human brain, which is strongly associated with AD status.…”

“…Firstly, we do not assume anymore that y 0 and m 0 are independent, as this does not hold in many medical imaging domains, e.g. for OA [34]. Namely, in the current version of CLIMAT, both the blocks R and P have now been allowed to make diagnosis predictions simultaneously, making sure that the learned embeddings contain information on y 0 .…”

Classification of Knee X-Ray Images by Severity of Osteoarthritis Using Skip Connection Based ResNet101

“…For this kind of reason, doctors reduce the occurrence of the disease [9]. In deciding this initial diagnosis technique is essential to identify the severe knee OA [10]. Some models are utilized proficient techniques like nuclear, MRI, X-ray, CT, International Journal of Intelligent Engineering and Systems, Vol.…”