Crowd-Sourced Verification of Computational Methods and Data in Systems Toxicology: A Case Study with a Heat-Not-Burn Candidate Modified Risk Tobacco Product

Abstract

Systems toxicology intends to quantify the effect of toxic molecules in biological systems and unravel their mechanisms of toxicity. The development of advanced computational methods is required for analyzing and integrating high throughput data generated for this purpose as well as for extrapolating predictive toxicological outcomes and risk estimates. To ensure the performance and reliability of the methods and verify conclusions from systems toxicology data analysis, it is important to conduct unbiased evaluations by independent third parties. As a case study, we report here the results of an independent verification of methods and data in systems toxicology by crowdsourcing. The sbv IMPROVER systems toxicology computational challenge aimed to evaluate computational methods for the development of blood-based gene expression signature classification models with the ability to predict smoking exposure status. Participants created/trained models on blood gene expression data sets including smokers/mice exposed to 3R4F (a reference cigarette) or noncurrent smokers/Sham (mice exposed to air). Participants applied their models on unseen data to predict whether subjects classify closer to smoke-exposed or nonsmoke exposed groups. The data sets also included data from subjects that had been exposed to potential modified risk tobacco products (MRTPs) or that had switched to a MRTP after exposure to conventional cigarette smoke. The scoring of anonymized participants' predictions was done using predefined metrics. The top 3 performers' methods predicted class labels with area under the precision recall scores above 0.9. Furthermore, although various computational approaches were used, the crowd's results confirmed our own data analysis outcomes with regards to the classification of MRTP-related samples. Mice exposed directly to a MRTP were classified closer to the Sham group. After switching to a MRTP, the confidence that subjects belonged to the smoke-exposed group decreased significantly. Smoking exposure gene signatures that contributed to the group separation included a core set of genes highly consistent across teams such as AHRR, LRRN3, SASH1, and P2RY6. In conclusion, crowdsourcing constitutes a pertinent approach, in complement to the classical peer review process, to independently and unbiasedly verify computational methods and data for risk assessment using systems toxicology.

The sbv IMPROVER Systems Toxicology computational challenge: Identification of human and species-independent blood response markers as predictors of smoking exposure and cessation status

“…Applying a human blood-based smoking exposure response gene signature previously identified [27], Martin et al demonstrated that samples from conventional cigarette smokers who switched to a THS2.2 shifted away from the S group and classified closer to the NCS group after only 5 days of switching [49]. These results were independently confirmed by the crowd in the context of this challenge [31]. …”

“…Crowdsourcing is a powerful approach to solve scientific problems, and also to independently verify methods, results, and conclusions [31]. Systems toxicology is an emerging discipline that requires the development of advanced computational methods for the analysis of large-scale datasets and the extrapolation of predictive toxicological outcomes and risk estimates.…”

“…The study design (Supplementary figure 1b) included five groups of mice that were exposed to: (1) cigarette smoke from a reference cigarette (3R4F), (2) air after 2-month exposure to 3R4F (Cess), (3) air only (Sham), (4) mainstream aerosol from a potential MRTP (pMRTP), or (5) switched to a pMRTP after 2-month exposure to 3R4F (Switch) [24]. Data from the latter two groups were provided to the challenge participants for verification purposes and results associated with these data are discussed in a separate manuscript [31]. For each group, blood samples were collected from 7 to 10 animals at 2, 3, 5 and 7 months.…”

“…Additional datasets provided for verification purposes [31] and not used for scoring were the H3/H4 (REXC-03-EU and C-04-JP) datasets for SC1, and the H3/H4 and M1b/M2b datasets for SC2. For each sub-challenge, test and verification sample data were randomized and then split into two subsets released sequentially at different dates during the challenge (Figure 1b).…”

Tobacco exposure-related alterations in DNA methylation and gene expression in human monocytes: the Multi-Ethnic Study of Atherosclerosis (MESA)

“…A number of previous studies have systematically investigated smoking-associated DNA methylation [ 5–12 ] and gene expression [ 13–15 ] profiles in blood samples. Alterations consistently observed in blood samples from smokers compared to non-smokers include DNA methylation profiles—within the aryl hydrocarbon receptor repressor ( AHRR , cg05575921), on chromosome 6p21.33 (cg06126421), and within coagulation factor II (thrombin) receptor-like 3 ( F2RL3 , cg03636183) [ 7 , 11 , 12 , 16 ]—and expression profiles of genes such as leucine rich repeat neuronal 3 ( LRRN3 ) and SAM and SH3 domain containing 1 ( SASH1 ) [ 13–15 , 17 , 18 ]. Smoking-associated DNA methylation profiles in blood have been used as biomarkers to improve risk prediction of smoking-related morbidity and mortality, including lung cancer incidence and mortality [ 19–21 ], cardiovascular mortality, and all-cause mortality [ 22 ].…”