Assessment of PLSDA cross validation

Abstract

Classifying groups of individuals based on their metabolic profile is one of the main topics in metabolomics research. Due to the low number of individuals compared to the large number of variables, this is not an easy task. PLSDA is one of the data analysis methods used for the classification. Unfortunately this method eagerly overfits the data and rigorous validation is necessary. The validation however is far from straightforward. Is this paper we will discuss a strategy based on cross model validation and permutation testing to validate the classification models. It is also shown that too optimistic results are obtained when the validation is not done properly. Furthermore, we advocate against the use of PLSDA score plots for inference of class differences.

Cordyceps sinensis protects against liver and heart injuries in a rat model of chronic kidney disease: a metabolomic analysis

“…A cluster of 500 permutated models from the first component was visualized using validation plots (Figures 3B, 3D). In the permutation test plots, all of the permutated Q2 values to the left were www.nature.com/aps Liu X et al Acta Pharmacologica Sinica npg lower than the original point to the right [39,40] , which indicated that the original models were valid. Moreover, according to coefficient numbers (|r|), VIP and P values, the major discriminatory metabolites for each group were determined in the pair-wise comparison.…”

Metabolic responses in gills of Manila clam Ruditapes philippinarum exposed to copper using NMR-based metabolomics

“…In PLS-DA, the X matrix is the measured matrix, i.e., the NMR data, and the Y matrix is composed of dummy variables (represented by ones and zeros) that indicate the class for each treatment . The prediction accuracy of the PLS-DA model was assessed using cross-validation with leave-one-out (Rubingh et al, 2006;Westerhuis et al, 2008). A Q 2 score > 0.08 indicates that the model classification is significantly better than chance, while a score greater than 0.4 indicates that the model is practically robust (Lindon et al, 1999).…”

Metabolic profiling reveals disorder of amino acid metabolism in four brain regions from a rat model of chronic unpredictable mild stress

“…Furthermore, a more sophisticated OPLS-DA model was achieved through removing the variation in X matrix unrelated to Y matrix so that the specific discriminant information between classes can be interpreted using one predictive component alone [34,37]. Great efforts have been made to test the reliability of multivariate models [38,39], hence the 6-round cross-validation in SMICA-P software was herein applied to validate the OPLS-DA model against over-fitting by precluding 1/6th of all the samples in each round. The crossvalidated OPLS-DA scores map depicts the between-class separation (t p ) and predictive ability (Q 2 Y) simultaneously [32].…”