Impartial graphical comparison of multivariate calibration methods and the harmony/parsimony tradeoff

Abstract

For multivariate calibration with the relationship y ¼ Xb, it is often necessary to determine the degrees of freedom for parsimony consideration and for the error measure root mean square error of calibration (RMSEC). This paper shows that degrees of freedom can be estimated by an effective rank (ER) measure to estimate the model fitting degrees of freedom and the more parsimonious model has the smallest ER. This paper also shows that when such a measure is used on the X-axis, simultaneous graphing of model errors and other regression diagnostics is possible for ridge regression (RR), partial least squares (PLS) and principal component regression (PCR) and thus, a fair comparison between all potential models can be accomplished. The ER approach is general and applicable to other multivariate calibration methods. It is often noted that by selecting variables, more parsimonious models are obtained; typically by multiple linear regression (MLR). By using the ER, the more parsimonious model is graphically shown to not always be the MLR model. Additionally, a harmony measure is proposed that expresses the bias/variance tradeoff for a particular model. By plotting this new measure against the ER, the proper harmony/parsimony tradeoff can be graphically assessed for RR, PCR and PLS. Essentially, pluralistic criteria for fairly valuating and characterizing models are better than a dualistic or a single criterion approach which is the usual tactic. Results are presented using spectral, industrial and quantitative structure activity relationship (QSAR) data.

Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters

“…The procedure based on sum of ranking differences (SRD) agrees well with multi-criteria decision making as it provides very similar rankings for models if the performance merits are used for SRD-based comparison (particularly when considering good models, as in Case study 1).…”

“…A biased model provides less variance and vice versa. However, harmonious models are not necessarily parsimonious [1]. The scope of the methodology has recently been extended with the idea of sum of ranking differences (SRD) for partial least squares and ridge regression models [2].…”

Sum of ranking differences (SRD) to ensemble multivariate calibration model merits for tuning parameter selection and comparing calibration methods

“…The minimization expression for the TR variant RR [24,[42][43][44] The L-curve for selecting tuning parameters [3,20,21,[24][25][26][27]29] can be formed by plotting mean RMSEC or RMSECV against a model variance or complexity measure. Models in the corner region of the L-curve represent acceptable compromises for the bias/variance tradeoff,…”

Effect of the subsampling ratio in the application of subagging for multivariate calibration with the successive projections algorithm

“…Sensitivity to instrumental noise was taken into account as suggested elsewhere [20][21][22] by calculating the 2-norm of the regression vector (||b av ||), which is defined as: (12) where b k av denotes the regression coefficient associated to variable x k in the ensemble model. In fact, by following the demonstration provided by Pinto et al, 20 it can be shown that s ŷ av = s noise ||b av ||, where s noise is the standard deviation of the instrumental noise (assumed to be homoscedastic and uncorrelated across the model variables) and s ŷ av is the standard deviation of the error in the ensemble model predictions resulting from the propagation of the instrumental noise.…”