Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies

Abstract

Partial Least Squares-Discriminant Analysis (PLS-DA) is a PLS regression method with a special binary ‘dummy’ y-variable and it is commonly used for classification purposes and biomarker selection in metabolomics studies. Several statistical approaches are currently in use to validate outcomes of PLS-DA analyses e.g. double cross validation procedures or permutation testing. However, there is a great inconsistency in the optimization and the assessment of performance of PLS-DA models due to many different diagnostic statistics currently employed in metabolomics data analyses. In this paper, properties of four diagnostic statistics of PLS-DA, namely the number of misclassifications (NMC), the Area Under the Receiver Operating Characteristic (AUROC), Q2 and Discriminant Q2 (DQ2) are discussed. All four diagnostic statistics are used in the optimization and the performance assessment of PLS-DA models of three different-size metabolomics data sets obtained with two different types of analytical platforms and with different levels of known differences between two groups: control and case groups. Statistical significance of obtained PLS-DA models was evaluated with permutation testing. PLS-DA models obtained with NMC and AUROC are more powerful in detecting very small differences between groups than models obtained with Q2 and Discriminant Q2 (DQ2). Reproducibility of obtained PLS-DA models outcomes, models complexity and permutation test distributions are also investigated to explain this phenomenon. DQ2 and Q2 (in contrary to NMC and AUROC) prefer PLS-DA models with lower complexity and require higher number of permutation tests and submodels to accurately estimate statistical significance of the model performance. NMC and AUROC seem more efficient and more reliable diagnostic statistics and should be recommended in two group discrimination metabolomic studies.Electronic supplementary materialThe online version of this article (doi:10.1007/s11306-011-0330-3) contains supplementary material, which is available to authorized users.

Metabolomic does not predict response to cardiac resynchronization therapy in patients with heart failure

“…The accuracy for classification was assessed by means of a double cross-validation scheme. 22,23 The original data set was split into a training set (80%) and a test set (20%) randomly before any step of statistical analysis. The number of OPLS components (3-40 components) was chosen on the basis of a five-fold cross-validation performed on the training set only, and the best model was used to predict the samples in the test set.…”

Metabolomics differentiation of oil palm (Elaeis guineensis Jacq.) spear leaf with contrasting susceptibility to Ganoderma boninense

“…However, no clear separation was obtained to differentiate the two groups of the oil palms, therefore, further separation of the groups were performed by supervised analysis of partial least squares-discriminant analysis (PLS-DA) and orthogonal partial least squares-discriminant analysis (OPLS-DA) model. PLS-DA regression, also known as projection on latent structures is a method practically used for classification a set of group samples and selection of biomarker in metabolomics studies (Szymańska et al, 2012). The model is built between dependent variables (Y) and independent variables (X) (i.e.…”

Direct infussion Electrospray Mass Spectrometry as a new non-invasive tool for serum metabolomics in induced-stress subjects

“…These groups of results were evaluated statistically and the resulting models showed values in line with quality parameters R 2 and Q 2 (explained variance of approximately 99% and a predicted variance above 50%) 53 . T-test analysis with p-values and data modelling using the PLS (Principal Least Squares) progression were carried out 54,55 . A large number of signals could be studied in the discrimination of classes considering the Variable-Importance-in-Projection-(VIP) which was set at a minimum value of 2.…”