Developing scientific confidence in HTS-derived prediction models: Lessons learned from an endocrine case study

Abstract

High throughput (HTS) and high content (HCS) screening methods show great promise in changing how hazard and risk assessments are undertaken, but scientific confidence in such methods and associated prediction models needs to be established prior to regulatory use. Using a case study of HTS-derived models for predicting in vivo androgen (A), estrogen (E), thyroid (T) and steroidogenesis (S) endpoints in endocrine screening assays, we compare classification (fitting) models to cross validation (prediction) models. The more robust cross validation models (based on a set of endocrine ToxCast™ assays and guideline in vivo endocrine screening studies) have balanced accuracies from 79% to 85% for A and E, but only 23% to 50% for T and S. Thus, for E and A, HTS results appear promising for initial use in setting priorities for endocrine screening. However, continued research is needed to expand the domain of applicability and to develop more robust HTS/HCS-based prediction models prior to their use in other regulatory applications. Based on the lessons learned, we propose a framework for documenting scientific confidence in HTS assays and the prediction models derived therefrom. The documentation, transparency and the scientific rigor involved in addressing the elements in the proposed Scientific Confidence Framework could aid in discussions and decisions about the prediction accuracy needed for different applications.

Weight-of-the-evidence evaluation of 2,4-D potential for interactions with the estrogen, androgen and thyroid pathways and steroidogenesis

“…As noted previously, a recent analysis by Cox et al (2014) shows good concordance between the ToxCast TM results and EDSP endpoints indicative for potential ER interaction. ToxCast TM assays showed no ER binding or transactivation potential for 2,4-D.…”

“…Although the thyroid pathway-related ToxCast TM assays for 2,4-D were negative, it should be noted that EPA has recently concluded that the ToxCast TM in vitro thyroid assays are not predictive of all relevant thyroid modes of Table 3. Results from EDSP in vitro studies of 2,4-D (data from individual study reports; published in Coady et al 2014 action in in vivo studies (Rotroff et al 2013), concordant with findings in Cox et al 2014 Ecotoxicological studies of 2,4-D relevant to assessment of potential endocrine pathway interactions…”

“…ToxCast TM assays showed no AR binding or transactivation potential for 2,4-D. Similar to findings for estrogen pathway-related activity, an analysis by Rotroff et al (2013) and a separate analysis by Cox et al (2014) showed relatively good concordance between the ToxCast TM results and EDSP Tier 1 endpoints indicative for potential AR interaction, including Hershberger assay results.…”

“…A more recent analysis (Cox et al 2014) using a case study of HTS-derived models for predicting in vivo androgen, estrogen and thyroid endpoints showed that the more robust cross validation models (based on a set of endocrine ToxCast TM assays and guideline in vivo endocrine screening studies) have balanced accuracies from 79 to 85% for androgen or estrogen pathway interactions, but predicted substantially less accuracy for thyroid endpoints.…”

Read-across approaches - misconceptions, promises and challenges ahead

“…Subsequently, efforts have been made to develop similar frameworks, with Blackburn and Stuard (2014) outlining a systematic uncertainty framework for developmental and reproductive toxicants (DART). In the meantime, the Cefic LRI organizing committee refocused its efforts into a read-across team and began to formulate its own scientific confidence framework taking into account the principles outlined by Cox et al (2014) for tox-21c approaches and their prediction models, as well as building on the work of Blackburn and Stuard (2014). A framework to characterize scientific confidence of being evaluated from earlier phase-in substance deadlines.…”