Large-scale benchmark study of survival prediction methods using multi-omics data

Abstract

Multi-omics data, that is, datasets containing different types of high-dimensional molecular variables, are increasingly often generated for the investigation of various diseases. Nevertheless, questions remain regarding the usefulness of multi-omics data for the prediction of disease outcomes such as survival time. It is also unclear which methods are most appropriate to derive such prediction models. We aim to give some answers to these questions through a large-scale benchmark study using real data. Different prediction methods from machine learning and statistics were applied on 18 multi-omics cancer datasets (35 to 1000 observations, up to 100 000 variables) from the database ‘The Cancer Genome Atlas’ (TCGA). The considered outcome was the (censored) survival time. Eleven methods based on boosting, penalized regression and random forest were compared, comprising both methods that do and that do not take the group structure of the omics variables into account. The Kaplan–Meier estimate and a Cox model using only clinical variables were used as reference methods. The methods were compared using several repetitions of 5-fold cross-validation. Uno’s C-index and the integrated Brier score served as performance metrics. The results indicate that methods taking into account the multi-omics structure have a slightly better prediction performance. Taking this structure into account can protect the predictive information in low-dimensional groups—especially clinical variables—from not being exploited during prediction. Moreover, only the block forest method outperformed the Cox model on average, and only slightly. This indicates, as a by-product of our study, that in the considered TCGA studies the utility of multi-omics data for prediction purposes was limited. Contact:moritz.herrmann@stat.uni-muenchen.de, +49 89 2180 3198 Supplementary information: Supplementary data are available at Briefings in Bioinformatics online. All analyses are reproducible using R code freely available on Github.

Integration strategies of multi-omics data for machine learning analysis

“…In that aspect, benchmark studies are also particularly useful and should be done more frequently. With the notable exception of Herrmann et al (2020) [168] which focused on survival prediction methods for multi-omics data, most benchmarks focus on clustering and dimensionality reduction methods [14] , [26] , [27] , [135] , [169] , [170] , [171] . Thorough comparisons of other ML models have not been made for multi-omics datasets, and we have yet to know if the deep learning prowess made in other fields of pattern recognition can be reproduced in bioinformatics [172] .…”

State of the Field in Multi-Omics Research: From Computational Needs to Data Mining and Sharing

“…A handful of notable multi-omics benchmarks are available, comparing: multi-omics and multi-view clustering algorithms ( Rappoport and Shamir, 2018 ), multi-omics dimensionality reduction ( Cantini et al, 2020 ) and multi-omics survival prediction methods ( Herrmann et al, 2020 ). All three benchmarks were performed using the TCGA cancer data.…”

“…(c) Packrat ( Packrat, 2020 ) (recently superseded by renv ) and checkpoint: dependency management packages specific to R, which help to create isolated and portable R environments. Checkpoint facilitated one of the previous multi-omics benchmarking efforts ( Herrmann et al, 2020 ).…”

DeepMOCCA: A pan-cancer prognostic model identifies personalized prognostic markers through graph attention and multi-omics data integration

“…These methods may also allow incorporation of background knowledge such as pathways or biological interaction networks which are crucial for understanding and representing cancer pathophysiology [20,21]. There are now a large number of methods for machine learning with multi-omics data [12,[22][23][24] using a wide range of different approaches; a common approach is the prediction of survival time for which benchmark datasets have been developed [25].…”