Current status and future opportunities in modeling Multiple Sclerosis clinical characteristics

Abstract

Development of effective treatments requires understanding of disease mechanisms. For diseases of the central nervous system (CNS), like Multiple sclerosis (MS), human pathology studies and animal models tend to identify candidate disease mechanisms. However, these studies cannot easily link identified processes to clinical outcomes, such as MS severity, required for causality assessment of candidate mechanisms. Technological advances now allow generation of thousands of biomarkers in living human subjects, derived from genes, transcripts, medical images and proteins or metabolites in biological fluids. These biomarkers can be assembled into computational models of clinical value, provided such models are generalizable. Reproducibility of models increases with technical rigor of study design, such as blinding, implementing controls, using large cohorts that encompass entire spectrum of disease phenotypes and, most importantly, validating models in independent cohort(s). To facilitate growth of this important research area, we performed a meta-analysis of publications that model MS clinical outcomes (n=302), extracting effect sizes, while also scoring technical quality of study design using pre-defined criteria. Finally, we generated a Shiny-App-based website that allows dynamic exploration of data using selective filtering. On average, published studies fulfilled only one out of seven criteria of study design rigor. Only 15.2% of studies used any validation strategy, and only 8% used the gold standard of independent cohort validation. Many studies also used small cohorts, e.g., for MRI and blood biomarker predictors the median sample size was below 100 subjects. We observed inverse relationships between reported effect sizes and the numbers of study design criteria fulfilled, expanding analogous reports from non-MS fields, that studies that fail to limit bias over-estimate effect sizes. In conclusion, the presented meta-analysis represents a useful tool for researchers, reviewers, and funders to improve design of future modeling studies in MS and to easily compare new studies with published literature. We expect that this will accelerate research in this important area, leading to development of robust models with proven clinical value.

Confounder-adjusted MRI-based predictors of multiple sclerosis disability

“…We have observed that cross-validation generates broad range of results that encompass the effect sizes of the independent validation cohort, but the median of cross-validation results is still overly optimistic. This is consistent with our extensive observations using independent validation in all our projects (Boukhvalova et al, 2019; Liu et al, 2022; Masvekar et al, 2021; Messan et al, 2022; Pham et al, 2021). Therefore, while cross-validation should be included in all modeling studies, the independent validation must be considered a gold standard.…”

“…Thanks to a recent meta-analysis of 302 papers describing models of MS clinical outcomes (Liu et al, 2022), we can compare our results with other published MRI biomarker-based models. Using an associated website that allow users to dynamically explore this rich dataset, we identified 40 papers that used MRI biomarkers to model EDSS as ordinal scale and reported p-values, and 20 papers that reported effect sizes as R^2.…”

“…Thanks to a recent meta-analysis of 302 papers describing models of MS clinical outcomes (Liu et al, 2022), we can compare our results with other published MRI biomarker-based models.…”

Molecular models of multiple sclerosis severity identify heterogeneity of pathogenic mechanisms

“…Consequently, our results are best interpreted in comparison to published literature. To do so, we recently published meta-analysis 45 of 302 publications that used clinical, imaging, or biomarker-based predictors of MS clinical outcomes: Table 2 of that meta-analysis summarizes studies predicting MS severity as continuous outcomes. The training cohorts' results explained maximum of 45% of variance, while independent validation cohorts explained maximum of 12% of variance.…”

Molecular mechanisms associated with multiple sclerosis progression, severity and phenotype

“…Based on the meta-analysis of MS models (28), these are the strongest validated effect sizes for these 3 traditional outcomes used for regulatory approval of MS drugs, using any type of predictors (i.e., clinical/demographic, MRI, blood/CSF biomarkers or genes).…”