Leveraging Uncertainty from Deep Learning for Trustworthy Material Discovery Workflows

Abstract

In this paper, we leverage predictive uncertainty of deep neural networks to answer challenging questions material scientists usually encounter in machine learning-based material application workflows. First, we show that by leveraging predictive uncertainty, a user can determine the required training data set size to achieve a certain classification accuracy. Next, we propose uncertainty-guided decision referral to detect and refrain from making decisions on confusing samples. Finally, we show that predictive uncertainty can also be used to detect out-of-distribution test samples. We find that this scheme is accurate enough to detect a wide range of real-world shifts in data, e.g., changes in the image acquisition conditions or changes in the synthesis conditions. Using microstructure information from scanning electron microscope (SEM) images as an example use case, we show that leveraging uncertainty-aware deep learning can significantly improve the performance and dependability of classification models.

Recent advances and applications of deep learning methods in materials science

“…Reaching high accuracy when training DL models implicitly assume the availability of a sufficiently large and diverse training dataset. Unfortunately, this rarely occurs in material discovery applications 288 . ML/DL models are prone to perform poorly on extrapolation 289 .…”

Explainable machine learning in materials science

“…One solution to this problem is to detect distribution shifts before trying to predict. Zhang et al 154 recently presented an interesting work in this direction. They leveraged predictive uncertainty from deep neural networks to detect real-world shifts in materials data (e.g.…”

Recent Advances and Applications of Deep Learning Methods in Materials Science

“…Reaching high accuracy when training DL models implicitly assumes the availability of a sufficiently large and diverse training dataset. Unfortunately, this rarely occurs in material discovery applications [333]. ML/DL models are prone to perform poorly on extrapolation [334] .…”

“…Careful evaluation of the uncertainty associated with DL predictions would not only increase reliability in predicted results but would also provide guidance on estimating the needed training data set size as well as suggesting what new data should be added to reach the target accuracy (uncertainty-guided decision). Zhang, Kailkhura, and Han's work emphasizes how including a UQ-motivated reject option into the DL model results in substantial improvements in the performance of the remaining material data [333]. Such a reject option is associated to the detection of out-of-distribution samples, which is only possible through UQ analysis of the predicted results.…”