Theory and Practice of Integrating Machine Learning and Conventional Statistics in Medical Data Analysis

Abstract

The practice of medical decision making is changing rapidly with the development of innovative computing technologies. The growing interest of data analysis with improvements in big data computer processing methods raises the question of whether machine learning can be integrated with conventional statistics in health research. To help address this knowledge gap, this paper presents a review on the conceptual integration between conventional statistics and machine learning, focusing on the health research. The similarities and differences between the two are compared using mathematical concepts and algorithms. The comparison between conventional statistics and machine learning methods indicates that conventional statistics are the fundamental basis of machine learning, where the black box algorithms are derived from basic mathematics, but are advanced in terms of automated analysis, handling big data and providing interactive visualizations. While the nature of both these methods are different, they are conceptually similar. Based on our review, we conclude that conventional statistics and machine learning are best to be integrated to develop automated data analysis tools. We also strongly believe that machine learning could be explored by health researchers to enhance conventional statistics in decision making for added reliable validation measures.

Predicting health insurance uptake in Kenya using Random Forest: An analysis of socio-economic and demographic factors

“…"Testing," on the other hand, involves evaluating the performance of the trained models on a separate subset of data that was not used during training. This allows us to assess how well the models generalize to new, unseen data [ 27 , 28 ].…”

Improving the Trustworthiness of Interactive Visualization Tools for Healthcare Data through a Medical Fuzzy Expert System

“…Dhillon et al examined in one of their works the conceptual integration of traditional statistics and AI, with a primary emphasis on the research undertaken in the field of health. They found that researchers in the field of medicine might look at using AI to supplement traditional statistical methods of decision making in order to obtain additional trustworthy validation metrics [13]. In the healthcare industry, analyzing the trustworthiness of performance helps doctors and other professionals reach their goals by pointing out actions that can help guide decision making, improve performance, and get them started on the path to security and excellence.…”