Development of effective treatments requires understanding of disease mechanisms. For diseases of the central nervous system (CNS), like Multiple sclerosis (MS), human pathology studies and animal models tend to identify candidate disease mechanisms. However, these studies cannot easily link identified processes to clinical outcomes, such as MS severity, required for causality assessment of candidate mechanisms. Technological advances now allow generation of thousands of biomarkers in living human subjects, derived from genes, transcripts, medical images and proteins or metabolites in biological fluids. These biomarkers can be assembled into computational models of clinical value, provided such models are generalizable. Reproducibility of models increases with technical rigor of study design, such as blinding, implementing controls, using large cohorts that encompass entire spectrum of disease phenotypes and, most importantly, validating models in independent cohort(s).
To facilitate growth of this important research area, we performed a meta-analysis of publications that model MS clinical outcomes (n=302), extracting effect sizes, while also scoring technical quality of study design using pre-defined criteria. Finally, we generated a Shiny-App-based website that allows dynamic exploration of data using selective filtering.
On average, published studies fulfilled only one out of seven criteria of study design rigor. Only 15.2% of studies used any validation strategy, and only 8% used the gold standard of independent cohort validation. Many studies also used small cohorts, e.g., for MRI and blood biomarker predictors the median sample size was below 100 subjects. We observed inverse relationships between reported effect sizes and the numbers of study design criteria fulfilled, expanding analogous reports from non-MS fields, that studies that fail to limit bias over-estimate effect sizes.
In conclusion, the presented meta-analysis represents a useful tool for researchers, reviewers, and funders to improve design of future modeling studies in MS and to easily compare new studies with published literature. We expect that this will accelerate research in this important area, leading to development of robust models with proven clinical value.
