Methodological issues in the design of a rheumatoid arthritis activity score and its cut-offs

Collignon, Olivier

doi:10.2147/clep.s64811

Cited by 6 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their multivariable analyses, Dr. Domsic and colleagues appropriately retained a limited number of predictors as allowed by the sample size of the study, i.e., 6 predictors for 110 patients who were deceased after 5 years versus 278 patients who were still living. These numbers are consistent with some previously published guidance on limiting model overfitting (2,3) when unpenalized likelihood is used to estimate the weights of the predictors. Moreover, despite the fact that the authors did not perform an internal validation to correct their findings from optimism (using, for example, bootstrap resampling techniques), they conducted a proper external validation by applying the constructed score to an independent data set.…”

Section: Statistical Considerations In the Development Of Clinical Prsupporting

confidence: 89%

Statistical Considerations in the Development of Clinical Predictive Scores: Comment on the Article by Domsic et al

Collignon

Veselý²

2016

Arthritis & Rheumatology

Self Cite

View full text Add to dashboard Cite

of combined pulmonary fibrosis and emphysema and idiopathic pulmonary fibrosis: over a 3-year follow-up. Tuberc Respir Dis (Seoul) We read with interest the article by Domsic et al (1), describing a study in which the authors aimed to build a predictive score for 5-year mortality in diffuse cutaneous systemic sclerosis (dcSSc). We would like to take the opportunity, using their article as an example, to discuss what could be considered good practice when developing predictive scores and what could probably improve the results of the analysis. We also would like to address the potential use of their score as an end point in clinical trials.In their multivariable analyses, Dr. Domsic and colleagues appropriately retained a limited number of predictors as allowed by the sample size of the study, i.e., 6 predictors for 110 patients who were deceased after 5 years versus 278 patients who were still living. These numbers are consistent with some previously published guidance on limiting model overfitting (2,3) when unpenalized likelihood is used to estimate the weights of the predictors. Moreover, despite the fact that the authors did not perform an internal validation to correct their findings from optimism (using, for example, bootstrap resampling techniques), they conducted a proper external validation by applying the constructed score to an independent data set. This is, in our view, a major asset of their study. This type of initiative remains, unfortunately, rare in published studies.In their analysis, Domsic et al obtained an area under the curve (AUC) of 69% in the independent validation data set. There are several methodologic approaches that could lead to an improvement of the model performance.First, the predictive performance of the model could gain accuracy by use of alternative predictor selection methods. Indeed, stepwise selection has been widely criticized over the past years because of its propensity to retain uninformative predictors in the model when the number of candidates is large, and to provide inflated estimates (biased high) of their coefficient (3) (the "beta weights" in the report by Domsic et al). Prefiltering the predictors according to their univariate significance, based on principles very similar to the first stage of forward variable selection, has also been discouraged, because this procedure discards predictors that could become important only after other specific ones have been included (4). As an alternative, the LASSO (least absolute shrinkage and selection operator) method described by Tibshirani (5) has been widely used in recent years to select small subsets of predictors, since it does not suffer from most of the drawbacks of stepwise selection.Also, although this is not an uncommon practice in medical publications discussing development of clinical scores, rounding the beta coefficients to an integer could lead to loss of precision of the predictive score and thus to a decreased AUC. The argument for simplifying a predictive score to render it easier to use in clinical pr...

show abstract

Section: Statistical Considerations In the Development Of Clinical Prsupporting

confidence: 89%

Statistical Considerations in the Development of Clinical Predictive Scores: Comment on the Article by Domsic et al

Collignon

Veselý²

2016

Arthritis & Rheumatology

Self Cite

View full text Add to dashboard Cite

show abstract

“…Table (7) and Figure (2) show that the difference between binucleated cells (BN) with MNi and BN cells with NPB and the necrotic and/or apoptotic cells between the 30 RA patients and the control group was statistically significant so illustrating that genotoxicity is a part of RA disease itself.…”

Section: Results Of Genotoxicity (Micronucleus Test)mentioning

confidence: 96%

“…It affects 1% of the world population and triggers joint inflammations that may worsen patient's quality of life 2 .…”

Section: Introductionmentioning

confidence: 99%

Assessment of Cytotoxicity and Genotoxicity in Patients with Rheumatoid Arthritis and its Relation to Disease Activity and Medications

Tayel¹,

Tayel²,

Mahrous

et al. 2015

Egypt J of Rheumatol & Clinical Immunol

View full text Add to dashboard Cite

show abstract

“…Second, three variants of bootstrap internal validation techniques were used: the regular bootstrap, the bootstrap .632 and the bootstrap .632+. These methods are described in details in Steyerberg et al [ 21 ] and Collignon [ 22 ]. All three methods use bootstrap samples of the original dataset (i.e.…”

Section: Methodsmentioning

confidence: 99%

Comparison of the modified unbounded penalty and the LASSO to select predictive genes of response to chemotherapy in breast cancer

Collignon

Han

et al. 2018

PLoS ONE

Self Cite

View full text Add to dashboard Cite

Covariate selection is a fundamental step when building sparse prediction models in order to avoid overfitting and to gain a better interpretation of the classifier without losing its predictive accuracy. In practice the LASSO regression of Tibshirani, which penalizes the likelihood of the model by the L1 norm of the regression coefficients, has become the gold-standard to reach these objectives. Recently Lee and Oh developed a novel random-effect covariate selection method called the modified unbounded penalty (MUB) regression, whose penalization function can equal minus infinity at 0 in order to produce very sparse models. We sought to compare the predictive accuracy and the number of covariates selected by these two methods in several high-dimensional datasets, consisting in genes expressions measured to predict response to chemotherapy in breast cancer patients. These comparisons were performed by building the Receiver Operating Characteristics (ROC) curves of the classifiers obtained with the selected genes and by comparing their area under the ROC curve (AUC) corrected for optimism using several variants of bootstrap internal validation and cross-validation. We found consistently in all datasets that the MUB penalization selected a remarkably smaller number of covariates than the LASSO while offering a similar—and encouraging—predictive accuracy. The models selected by the MUB were actually nested in the ones obtained with the LASSO. Similar findings were observed when comparing these results to those obtained in their first publication by other authors or when using the area under the Precision-Recall curve (AUCPR) as another measure of predictive performance. In conclusion, the MUB penalization seems therefore to be one of the best options when sparsity is required in high-dimension. Further investigation in other datasets is however required to validate these findings.

show abstract

Methodological issues in the design of a rheumatoid arthritis activity score and its cut-offs

Cited by 6 publications

References 29 publications

Statistical Considerations in the Development of Clinical Predictive Scores: Comment on the Article by Domsic et al

Statistical Considerations in the Development of Clinical Predictive Scores: Comment on the Article by Domsic et al

Assessment of Cytotoxicity and Genotoxicity in Patients with Rheumatoid Arthritis and its Relation to Disease Activity and Medications

Comparison of the modified unbounded penalty and the LASSO to select predictive genes of response to chemotherapy in breast cancer

Contact Info

Product

Resources

About