Repeated double cross validation

Filzmoser, Peter; Liebmann, B.; Вармуза, Курт

doi:10.1002/cem.1225

Cited by 412 publications

(334 citation statements)

References 28 publications

Supporting

Mentioning

320

Contrasting

Unclassified

Order By: Relevance

“…While each time one fold was withheld in the outer loop as an independent test set (test fold) to compute unbiased estimates of prediction performance, parameter selection was carried out by 4-fold inner cross-validation on the remaining four folds (training folds). Once the best parameters had been found for the four training folds of the outer cross-validation loop, a model was trained on these four folds using the optimized parameters and applied to the withheld test fold to estimate the prediction performance on independent data (45,46). This was applied to all five possible combinations of alternating splits (5-fold cross-validation), finally resulting in five sets of markers for any tested condition.…”

Section: Methodsmentioning

confidence: 99%

“…The P-SVM has been designed for exactly such purposes and has proven to be highly successful for analyzing high-dimensional molecular data (56). The training of the models, along with the selection of optimal hyperparameters, was done by nested cross-validation in order to facilitate model selection while still obtaining unbiased estimates of the generalization performance (45,46).…”

Section: Figmentioning

confidence: 99%

See 1 more Smart Citation

Transcriptome Profiling of Antimicrobial Resistance in Pseudomonas aeruginosa

Khaledi

Schniederjans

Pohl

et al. 2016

Antimicrob Agents Chemother

View full text Add to dashboard Cite

e Emerging resistance to antimicrobials and the lack of new antibiotic drug candidates underscore the need for optimization of current diagnostics and therapies to diminish the evolution and spread of multidrug resistance. As the antibiotic resistance status of a bacterial pathogen is defined by its genome, resistance profiling by applying next-generation sequencing (NGS) technologies may in the future accomplish pathogen identification, prompt initiation of targeted individualized treatment, and the implementation of optimized infection control measures. In this study, qualitative RNA sequencing was used to identify key genetic determinants of antibiotic resistance in 135 clinical Pseudomonas aeruginosa isolates from diverse geographic and infection site origins. By applying transcriptome-wide association studies, adaptive variations associated with resistance to the antibiotic classes fluoroquinolones, aminoglycosides, and ␤-lactams were identified. Besides potential novel biomarkers with a direct correlation to resistance, global patterns of phenotype-associated gene expression and sequence variations were identified by predictive machine learning approaches. Our research serves to establish genotype-based molecular diagnostic tools for the identification of the current resistance profiles of bacterial pathogens and paves the way for faster diagnostics for more efficient, targeted treatment strategies to also mitigate the future potential for resistance evolution.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Figmentioning

confidence: 99%

Transcriptome Profiling of Antimicrobial Resistance in Pseudomonas aeruginosa

Khaledi

Schniederjans

Pohl

et al. 2016

Antimicrob Agents Chemother

View full text Add to dashboard Cite

show abstract

“…The leave-one-out (LOO) crossvalidation method [29][30][31][32] was used to find out the optimal set of spectral lines and to estimate the regression model predictivity. We used spectra of 36 pellets with nine different heating values for calibration, and for testing, we left out a subset of spectra of four pellets of the same sample having the known heating value from calorimetric measurements.…”

Section: Cross-validationmentioning

confidence: 99%

Determination of Heating Value of Estonian Oil Shale by Laser-Induced Breakdown Spectroscopy

Aints¹,

Paris²,

Laan³

et al. 2018

Journal of Spectroscopy

View full text Add to dashboard Cite

The laser-induced breakdown spectroscopy (LIBS) combined with multivariate regression analysis of measured data were utilised for determination of the heating value and the chemical composition of pellets made from Estonian oil shale samples with different heating values. The study is the first where the oil shale heating value is determined on the basis of LIBS spectra. The method for selecting the optimal number of spectral lines for ordinary multivariate least squares regression model is presented. The correlation coefficient between the heating value predicted by the regression model, and that measured by calorimetric bomb, was R 2 = 0 98. The standard deviation of prediction was 0.24 MJ/kg. Concentrations of oil shale components predicted by the regression model were compared with those measured by ordinary methods.

show abstract

“…Many authors favor cross-validation as it 'gives a reliable picture with no apparent systematic over-or underestimation' [31]; 'overfitting is avoided by the repeated double cross-validation approach' [32]; 'LOO gives too small a perturbation to the data, so that Q 2 approaches the properties of R 2 asymptotically' [33]; 'The crossvalidation estimate of prediction error is nearly unbiased but can be highly variable.' [34].…”

Section: Use Of Alternative Training-test Set Splitsmentioning

confidence: 99%

Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters

Rácz

Bajusz

Héberger

2015

SAR and QSAR in Environmental Research

View full text Add to dashboard Cite

Recent implementations of QSAR modeling software provide the user with numerous models and a wealth of information. In this work, we provide some guidance on how one should interpret the results of QSAR modeling, compare and assess the resulting models and select the best and most consistent ones. Two QSAR datasets are applied as case studies for the comparison of model performance parameters and model selection methods. We demonstrate the capabilities of sum of ranking differences (SRD) in model selection and ranking and identify the best performance indicators and models. While the exchange of the original training and (external) test sets does not affect the ranking of performance parameters, it provides improved models in certain cases (despite the lower number of molecules in the training set). Performance parameters for external validation are substantially separated from the other merits in SRD analyses, highlighting their value in data fusion. IntroductionModel comparison and selection of the best one is an evergreen among scientific investigations. The process is contradictory: bias-variance trade-off, local minima, searching for robust models, the principle of parsimony, etc.; all ideas consider various models inherently. One model is better from one point of view, the other should be better from another point of view. Even if one fixes the aim (and algorithm) according to various criteria: R 2 , Q 2 , Mallows Cp, Akaike Information criterion, Bayesian information criterion, etc., their application on the training, validation and test sets will necessarily provide different models for description of existing data and for prediction of future samples. The case is even more complicated with the fact that we deal with random effects: i.e. it is relatively easy to find conditions where one of the models is clearly superior compared to other models. Many authors select instinctively or deliberately such datasets, splits, etc. for which their own descriptor selection or model building algorithm performs better than the rival approaches.Kalivas et al. suggested selecting harmonious models taking into account the bias-variance trade-off: it is difficult and not unambiguous to find the 'best' model. A biased model provides less variance and vice versa. However, harmonious models are not necessarily parsimonious [1]. The scope of the methodology has recently been extended with the idea of sum of ranking differences (SRD) for partial least squares and ridge regression models [2].Principal-component analysis (PCA) has been applied by Geladi [3,4] and Todeschini et al. [5] to find the best and worst regression and classification models, respectively. PCAs were completed on a matrix of regression vectors and dominant patterns (grouping, outliers) could be detected among the models. The interpretation of PCA results is easy: principal component 1 marks the direction of the best and worst regression models. Principal component 2 reflects various behaviors of the regression models on various datasets. The models lyin...

show abstract

Repeated double cross validation

Cited by 412 publications

References 28 publications

Transcriptome Profiling of Antimicrobial Resistance in Pseudomonas aeruginosa

Transcriptome Profiling of Antimicrobial Resistance in Pseudomonas aeruginosa

Determination of Heating Value of Estonian Oil Shale by Laser-Induced Breakdown Spectroscopy

Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters

Contact Info

Product

Resources

About