Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

Maddouri, Omar; Qian, Xiaoning; Alexander, Francis J.; Dougherty, Edward R.; Yoon, Byung-Jun

doi:10.1016/j.patter.2021.100428

“…By assigning appropriate weights to each sample drawn from the importance distribution, Importance Sampling can adjust the synthetic data points, giving more significance to those that align well with the target population's distribution. In recent work [44], Importance Sampling helped handle uncertainty in small-sample scenarios during classification tasks by selecting and weighting data points from an alternative distribution. This approach enabled more accurate estimation of classification errors, providing a robust and reliable assessment of classifier performance, even in situations with limited training data.…”

Section: Addressing Data Biasmentioning

confidence: 99%

High-Fidelity Synthetic Data Applications for Data Augmentation

Wang,

Draghi,

Rotalinti

et al. 2024

Artificial Intelligence

3

0

View full text Add to dashboard Cite

The use of high-fidelity synthetic data for data augmentation is an area of growing interest in data science. In this chapter, the concept of synthetic data is introduced, and different types of synthetic data are discussed in terms of their utility or fidelity. Approaches to synthetic data generation are presented and compared with computer modelling and simulation approaches, highlighting the unique benefits of high-fidelity synthetic data. One of the main applications of high-fidelity synthetic data is supporting the training and validation of machine learning algorithms, where it can provide a virtually unlimited amount of diverse and high-quality data to improve the accuracy and robustness of models. Furthermore, high-fidelity synthetic data can address missing data and biases due to under-sampling using techniques such as BayesBoost, as well as boost sample sizes in scenarios where the real data is based on a small sample. Another important application is generating virtual patient cohorts, such as digital twins, to estimate counterfactuals in silico trials, allowing for better prediction of treatment outcomes and personalised medicine. The chapter concludes by identifying areas for further research in the field, including developing more efficient and accurate synthetic data generation methods and exploring the ethical implications of using synthetic data.

show abstract

“…This section review model selection in the context of error estimation under all settings i.e., traditional machine learning, domain adaption and transfer learning. (Maddouri et al, 2022)…”

Section: Related Workmentioning

confidence: 99%

“…In parametric methods, there is the popular plug-in estimator, which naively estimates the true error from an empirical model (Maddouri et al, 2022). A major drawback of this approach is that it is strongly dependent on parameter estimation, which may lead to catastrophic failures.…”

Section: Parametric Error Estimationmentioning

confidence: 99%

Improving Model Selection in Deep Supervised Transfer Learning Under Homogeneous Setting

Omondiagbe¹,

MacDonell²,

Licorish³

2022

Preprint

0

View full text Add to dashboard Cite

In traditional machine learning environments, the use of non-parametric error estimation to set the discriminative threshold of a classifier to achieve the best accuracy is very effective. This method is not effective in a transfer learning environment because it is only reliable when both the training and testing data have similar distributions which is not the case in a transfer learning setting. Although the use of control variate techniques has been proposed to exploit the information about the error in the training sample to reduce the error in the test sample, this method yields a finite variance and the model uncertainty is not distributed among the variance. In this paper, we proposed and test a new transfer learning validation method called control linear minimum mean-squared error (CLMMSE) for source model selection under homogeneous transfer learning settings with the absence of an adequate pre-trained source model. Our approach adopts the Bayesian linear minimum mean-squared error (LMMSE) and integrates the idea of importance sampling into a control variate approach to provide an accurate estimate for the LMMSE that is then used to select the optimal source model. By combining importance sampling with the control variate technique to reduce further the variance, we can achieve a much tighter bound with the LMMSE. This approach reduces the risk in the target domain under data shift. Experimental results on synthetic data under two data shift settings demonstrate the efficacy of our approach. A further experiment on two real-world datasets shows that we were able to improve the accuracy of two state-of-the-art models tested; Bert (0.94\% to 65\%) and CodeBERT (1.82\% to 18.2\%) when compared to using previous selection methods.

show abstract

“…There are two broad categories of error or validation estimation schemes: parametric and non-parametric (Maddouri et al, 2022;Liu and Yang, 2011). Non-parametric estimates compute the error rate by counting the misclassified points, with widely used estimators which includes re-substitution, crossvalidation (CV) and bootstrap estimators.Parametric estimates the true error from an empirical model.…”

Section: Error Estimationmentioning

confidence: 99%

Improving Model Selection in Deep Supervised Transfer Learning Under Homogeneous Setting

Omondiagbe¹,

MacDonell²,

Licorish³

2022

Preprint

0

View full text Add to dashboard Cite

In traditional machine learning environments, the use of non-parametric error estimation to set the discriminative threshold of a classifier to achieve the best accuracy is very effective. This method is not effective in a transfer learning environment because it is only reliable when both the training and testing data have similar distributions which is not the case in a transfer learning setting. Although the use of control variate techniques has been proposed to exploit the information about the error in the training sample to reduce the error in the test sample, this method yields a finite variance and the model uncertainty is not distributed among the variance. In this paper, we proposed and test a new transfer learning validation method called control linear minimum mean-squared error (CLMMSE) for source model selection under homogeneous transfer learning settings with the absence of an adequate pre-trained source model. Our approach adopts the Bayesian linear minimum mean-squared error (LMMSE) and integrates the idea of importance sampling into a control variate approach to provide an accurate estimate for the LMMSE that is then used to select the optimal source model. By combining importance sampling with the control variate technique to reduce further the variance, we can achieve a much tighter bound with the LMMSE. This approach reduces the risk in the target domain under data shift. Experimental results on synthetic data under two data shift settings demonstrate the efficacy of our approach. A further experiment on two real-world datasets shows that we were able to improve the accuracy of two state-of-the-art models tested; Bert (0.94\% to 65\%) and CodeBERT (1.82\% to 18.2\%) when compared to using previous selection methods.

show abstract

Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

Cited by 4 publications

References 26 publications

High-Fidelity Synthetic Data Applications for Data Augmentation

High-Fidelity Synthetic Data Applications for Data Augmentation

Improving Model Selection in Deep Supervised Transfer Learning Under Homogeneous Setting

Improving Model Selection in Deep Supervised Transfer Learning Under Homogeneous Setting

Contact Info

Product

Resources

About