SurvBoard: Standardised Benchmarking for Multi-omics Cancer Survival Models

Abstract

High-throughput "omics" data, including genomic, transcriptomic, and epigenetic data, have become increasingly produced and have contributed in recent years to the advances in cancer research. In particular, multimodal omics data get now employed in addition to clinical data to stratify patients according to their clinical outcomes. Despite some recent work on benchmarking multi-modal integration strategies for cancer survival prediction, there is still a need for the standardization of the results of model performances and for the consecutive exploration of the relative performance of statistical and deep learning models. Here, we propose a unique benchmark, SurvBoard, which standardizes several important experimental design choices to enable comparability between cancer survival models that incorporate multi-omics data. By designing several benchmarking scenarios, SurvBoard allows for the comparison of single-cancer models and models trained on pan-cancer data; SurvBoard also makes it possible to investigate the added value of using patient data with missing modalities. Additionally, in this work, we point out several potential pitfalls that might arise during the preprocessing and validation of multi-omics cancer survival models and address them in our benchmark. We compare statistical and deep learning models revealing that statistical models often outperform deep learning models, particularly in terms of model calibration. Finally, we offer a web service that enables quick model evaluation against our benchmark (https://www.survboard.science/). All code and other resources are available on GitHub: https://github.com/BoevaLab/survboard/.

Survival Prediction Landscape: An In-Depth Systematic Literature Review on Activities, Methods, Tools, Diseases, and Databases

“…For instance, it hinders the establishment of standardized testing and benchmarking procedures for newly proposed survival prediction methods, leading to ambiguities in identifying the most advanced techniques. Moreover, recognizing the need for standardization in benchmarking survival prediction models, Wissel et al 58 introduced benchmark survival datasets tailored for both individual cancer subtypes and pancancer settings. These datasets are accessible at https://survboard.vercel.app/, contributing to a more uniform and transparent benchmarking framework within the survival prediction landscape.…”

“…In Table 1, a comprehensive analysis of the scope of review articles indicates that existing studies can be classified into three distinct groups. I) 9 review papers primarily focus on the application of DL algorithms in survival prediction 47,[49][50][51][52][53][54][55][56] , II) 7 review papers summarise the application of ML algorithms in survival prediction 37,48,[57][58][59][60][61] , and 6 review papers summarise survival prediction methods from three different categories namely statistical, ML, and DL methods [44][45][46][62][63][64] .…”

Survival prediction landscape: an in-depth systematic literature review on activities, methods, tools, diseases, and databases

“…The lack of shared data and presence of multiple datasets associated with a single disease pose a notable challenge in survival prediction. For instance, it hinders the establishment of standardized testing and benchmarking procedures for newly proposed survival prediction methods, leading to ambiguities in identifying the most advanced techniques (Wissel et al, 2022 ). Moreover, recognizing the need for standardization in benchmarking survival prediction models, Wissel et al ( 2022 ) introduced benchmark survival datasets tailored for both individual cancer subtypes and pan-cancer settings.…”

“…In Table 2 , a comprehensive analysis of the scope of review articles indicates that existing studies can be classified into three distinct groups. (I) Nine review papers primarily focus on the application of DL algorithms in survival prediction (Ahmed, 2005 ; Bakasa and Viriri, 2021 ; Kvamme and Borgan, 2021 ; Pobar et al, 2021 ; Kantidakis et al, 2022 ; Altuhaifa et al, 2023 ; Salerno and Li, 2023 ; Wekesa and Kimwele, 2023 ; Wiegrebe et al, 2023 ), (II) seven review papers summarize the application of ML algorithms in survival prediction (Gupta et al, 2018 ; Lee and Lim, 2019 ; Boshier et al, 2022 ; Guan et al, 2022 ; Mo et al, 2022 ; Wissel et al, 2022 ; Feldner-Busztin et al, 2023 ), andsix review papers summarize survival prediction methods from three different categories namely statistical, ML, and DL methods (Bashiri et al, 2017 ; Herrmann et al, 2021 ; Tewarie et al, 2021 ; Westerlund et al, 2021 ; Deepa and Gunavathi, 2022 ; Rahimi et al, 2023 ).…”

Comparative analysis of integrative classification methods for multi-omics data

“…Wissel et al. [ 27 ] confirmed this result on TCGA, but observed an opposite trend on ICGC and TARGET datasets where multi-omics data led to improved performance relative to clinical-only models. They further highlighted the superiority of penalized Cox model and random survival forest over neural networks in terms of model calibration.…”