Reliable Model Selection without Reference Values by Utilizing Model Diversity with Prediction Similarity

Abstract

Predictive modeling (calibration or training) with various data formats, such as near-infrared (NIR) spectra and quantitative structure−activity relationship (QSAR) data, provides essential information if a proper model is selected. Similarly, with a general model selection approach, spectral model maintenance (updating) from original modeling conditions to new conditions can be performed for dynamic modeling. Fundamental modeling (partial least-squares (PLS) and others) and maintenance processes (domain adaptation or transfer learning and others) require selection of tuning parameter(s) values to isolate models that can accurately predict new samples or molecules, e.g., number of PLS latent variables to predict analyte concentration. Regardless of the modeling task, model selection is complex and without a reliable protocol. Tuning parameter selection typically depends on only one model quality measure assessing model bias using prediction accuracy. Developed in this paper is a generic model selection process using concepts from consensus modeling and QSAR activity landscapes. It is a consensus filtering approach that prioritizes model diversity (MD) while conserving prediction similarity (PS) fused with a common bias-variance trade-off measure. A significant feature of MDPS is that a cross-validation scheme is not needed because models are selected relative to predicting new samples or molecules, i.e., model selection uses unlabeled samples (without reference values) for active predictions. The versatility and reliability of MDPS model selection is shown using four NIR data sets and a QSAR data set. The study also substantiates the Rashomon effect where there is not one best model tuning parameter value that provides accurate predictions.

Calibration Model Updating to Novel Sample and Measurement Conditions without Reference Values

“…Specifically, poor predicting models corresponding to the RMSEV color-coded points light blue and red are pushed out of the selection region with ω = 0.4. Another characterization of this effect is shown in ref .…”

“…As previously noted, only models in the diversity range 0.3 < cos­(θ) < 0.5 at ω = 0.4 were evaluated for possible model selection. This diversity range and the ω value were empirically determined optimal in a previous work involving two tuning parameters and small deviations from these values are acceptable …”

“…The MDPS method is briefly described and further details are provided in ref . After the entire model set is generated over a range of tuning parameter values, all possible combinations of two models are compared for diversity and prediction similarities.…”

“…Generally, the more tuning parameters to be determined, the larger the size of the model space spanning all the generated models and the greater the model diversity existing in this model space, e.g., a PLS set of models for a primary calibration with one tuning parameter (number of LVs) compared to the NAR family of models with two to three tuning parameters. These points are further discussed in ref in the framework of the Rashomon effect where there is not one best accurate predicting model, but a collection of models exists with similar accurate analyte predictions. , Additionally, the degree of domain difference between the primary and secondary conditions and the density of acceptable models across the tuning parameter values may affect the cos­(θ) and ω values.…”

“…Recently, a new approach to model selection with one or more tuning parameters was developed and evaluated . It is based on model diversity and prediction similarity (MDPS).…”