Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery

Abstract

Traditional machine learning (ML) metrics overestimate model performance for materials discovery. We introduce (1) leave-onecluster-out cross-validation (LOCO CV) and (2) a simple nearestneighbor benchmark to show that model performance in discovery applications strongly depends on the problem, data sampling, and extrapolation. Our results suggest that ML-guided iterative experimentation may outperform standard high-throughput screening for discovering breakthrough materials like high-T c superconductors with ML.Materials informatics (MI), or the application of data-driven algorithms to materials problems, has grown quickly as a field in recent years. 9 Across all of these applications, a training database of simulated or experimentally-measured materials properties serves as input to a ML algorithm that predictively maps features (i.e., materials descriptors) to target materials properties. Ideally, the result of training such models would be the experimental realization of new materials with promising properties. The MI community has produced several such success stories, including thermoelectric compounds, 10,11 shapememory alloys, 12 superalloys, 13 and 3d-printable high-strength aluminum alloys. 14 However, in many cases, a model is itself the output of a study, and the question becomes: to what extent could the model be used to drive materials discovery? Typically, the performance of ML models of materials properties is quantified via cross-validation (CV). CV can be performed either in a single division of the available data into a training set (to build the model) and a test set (to evaluate its performance), or as an ensemble process known as k-fold CV wherein the data are partitioned into k nonoverlapping subsets of nearly equal size (folds) and model performance is averaged across each combination of k-1 training folds and one test fold. Leave-one-out crossvalidation (LOOCV) is the limit where k is the number of total examples in the dataset. Table 1 summarizes some examples of model performance statistics as reported in the aforementioned studies (some studies involved testing multiple algorithms across multiple properties).In Table 1, the reported model performance is uniformly excellent across all studies. A tempting conclusion is that any of these models could be used for one-shot high-throughput screening of large numbers of materials for desired properties. However, as we discuss below, traditional CV has critical shortcomings in terms of quantifying ML model performance for materials discovery. Issues with traditional crossvalidation for materials discoveryMany ML benchmark problems consist of data classification into discrete bins, i.e., pattern matching. For example, the Design, System, ApplicationMachine learning (ML) has become a widely-adopted predictive tool for materials design and discovery. Random k-fold cross-validation (CV), the traditional gold-standard approach for evaluating the quality of ML models, is fundamentally mismatched to the nature of materials discovery, and leads to ...

A Critical Review of Machine Learning of Energy Materials

“…However, if only one group of superconductors was used as training data, the models failed to predict other groups, as shown in Figure 17b–e. This finding is consistent with a later study by Meredig et al, where the authors noted that the superconductivity data formed distinct groups and the error of the models on a given group would increase sharply if no training data from this group was included in the model. The conclusion seems to suggest that different mechanisms are at play for different superconductor groups and in fact is known in the community.…”

“…Many materials problems involve the search for materials that possess rare combinations of properties or extraordinary properties, while such materials may not exist in the available data space. Hence a highly accurate ML model model trained on K ‐fold CV may not generalize well to novel material classes …”

Recent advances and applications of machine learning in solid-state materials science

“…Different methods ranging from a simple holdout, over k-fold cross-validation, leave-one-out cross-validation, Monte Carlo crossvalidation, 72 up to leave-one-cluster-out cross-validation 73 can be used for the evaluation. All these methods rely on keeping some data hidden from the model during the training process.…”

“…While this can be advantageous, it also means that a sample is not guaranteed to be in the test/training set. Leave-onecluster-out cross-validation 73 was specifically developed for materials science and estimates the ability of the machine learning model to extrapolate to novel groups of materials that were not present in the training data. Depending on the target quantity, this allows for a more realistic evaluation and a better understanding of the limitations of the machine learning model.…”

Enumeration of de novo inorganic complexes for chemical discovery and machine learning

“…In recent years, machine learning (ML) property prediction models trained on first-principles simulation data have further accelerated this discovery process 4,[11][12][13][14][15][16][17][18] throughout chemistry, [19][20][21][22][23][24] including for catalysis 15,16,25,26 and materials. 4,[27][28][29][30][31][32][33][34] Unique challenges arise in applying these tools to the discovery of open shell transition metal complexes, despite their importance as selective catalysts [35][36][37][38][39][40][41][42][43] and functional materials (e.g., molecular switches or sensors [44][45][46][47][48][49][50][51][52] ). The theoretical chemical space of inorganic complexes is diverse and relatively unexplored due to the variable spin states, oxidation states, and coordination num...…”