Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach

Abstract

Any machine learning (ML) model is highly dependent on the data it uses for learning, and this is even more important in the case of deep learning models. The problem is a data bottleneck, i.e. the difficulty in obtaining an adequate number of cases and quality data. Another issue is improving the learning process, which can be done by actively introducing experts into the learning loop, in what is known as human-in-the-loop (HITL) ML. We describe an ML model based on a neural network in which HITL techniques were used to resolve the data bottleneck problem for the treatment of pancreatic cancer. We first augmented the dataset using synthetic cases created by a generative adversarial network. We then launched an active learning (AL) process involving human experts as oracles to label both new cases and cases by the network found to be suspect. This AL process was carried out simultaneously with an interactive ML process in which feedback was obtained from humans in order to develop better synthetic cases for each iteration of training. We discuss the challenges involved in including humans in the learning process, especially in relation to human–computer interaction, which is acquiring great importance in building ML models and can condition the success of a HITL approach. This paper also discusses the methodological approach adopted to address these challenges.

Exploring Data Augmentation and Active Learning Benefits in Imbalanced Datasets

“…A human-in-the-loop approach, combining a CTGAN for data augmentation and an active learning module for addressing data bottlenecks in medical deep learning models, has been proposed in [36]. The effectiveness of artificial data in active learning scenarios has also been studied in [37], by using G-SMOTE as an artificial data generator and introducing it into the traditional active learning framework in order to reduce the amount of labeled data required in active learning.…”

“…We chose a pool-based strategy due to the popularity of this approach. Since many different query strategies exist and the selection is not straightforward, we decided to use entropy sampling, as it is a well-known strategy used in typical active learning scenarios [36,41].…”

Improving Medical Data Annotation Including Humans in the Machine Learning Loop

“…Our goal in including human experts in the feature selection process is to improve the explanatory power of the resulting models. As described in [46], they could also be involved with the aim of obtaining models with a higher accuracy.…”

“…First of all, we can say that the dataset has few cases, so ML models suffer when trying to generalize patterns present in the data. This is a clear data bottleneck problem and, as discussed in [46], a possible solution is to use data augmentation strategies to improve data quality and quantity. In that work the accuracy increased more than 10 percent with the collaboration of human experts who helped to improve the labeling and the generation of synthetic cases.…”