Use and Misuse of Waterfall Plots

Shao, Tiffany; Wang, Lisa; Templeton, Arnoud J.; Jang, Raymond Woo-Jun; Vera-Badillo, Francisco W.; McNamara, Mairéad G.; Margolis, M.; Kim, Tae Kyoung; Sinaei, Mehrdad; Shoushtari, Hassan; Tannock, Ian F.

doi:10.1093/jnci/dju331

Cited by 18 publications

(12 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first is that in contrast with OS, where the date of death is precise and can be ascertained for all patients, surrogates are prone to reader interpretation, measurement error, evaluation bias, and attrition bias. 65,66 These artifacts may create spurious surrogate benefits. The second explanation is a biological one: a cancer drug with favorable surrogate effects may affect changes in tumor growth or aggressiveness or may increase off-target deaths.…”

Section: Discussionmentioning

confidence: 99%

Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration's Approval of Oncology Drugs

Kim

Prasad

2016

Mayo Clinic Proceedings

103

View full text Add to dashboard Cite

Objective To determine the strength of the surrogate-survival correlation for cancer drug approvals based on a surrogate. Participants and Methods We performed a retrospective study of the US Food and Drug Administration (FDA) database, with focused searches of MEDLINE and Google Scholar. Among cancer drugs approved based on a surrogate end point, we examined previous publications assessing the strength of the surrogate-survival correlation. Specifically, we identified the percentage of surrogate approvals lacking any formal analysis of the strength of the surrogate-survival correlation, and when conducted, the strength of such correlations. Results Between January 1, 2009, and December 31, 2014, the FDA approved marketing applications for 55 indications based on a surrogate, of which 25 were accelerated approvals and 30 were traditional approvals. We could not find any formal analyses of the strength of the surrogate-survival correlation in 14 out of 25 accelerated approvals (56%) and 11 out of 30 traditional approvals (37%). For accelerated approvals, just 4 approvals (16%) were made where a level 1 analysis (the most robust way to validate a surrogate) had been performed, with all 4 studies reporting low correlation (r≤0.7). For traditional approvals, a level 1 analysis had been performed for 15 approvals (50%): 8 (53%) reported low correlation (r≤0.7), 4 (27%) medium correlation (r>0.7 to r<0.85), and 3 (20%) high correlation (r≥0.85) with survival. Conclusions The use of surrogate end points for drug approval often lacks formal empirical verification of the strength of the surrogate-survival association.

show abstract

Section: Discussionmentioning

confidence: 99%

Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration's Approval of Oncology Drugs

Kim

Prasad

2016

Mayo Clinic Proceedings

103

View full text Add to dashboard Cite

show abstract

“…29,30 In addition, medical oncologists with limited training in imaging or application of imaging criteria may have high interobserver variability in their assessment compared with other oncologists and radiologists. 31 These alternate workflows for response assessment may lead to inefficiency, high variability, and low interreader agreement. The role of a single IA per trial can serve as a catalyst in the relationship between radiologists, oncologists, and clinical staff, enabling improved reliability, decreased interreader variability, and faster turnaround time without experimenter bias.…”

Section: Discussionmentioning

confidence: 99%

“…27,29,30 However, medical oncologists are often inadequately trained for both imaging assessment and the application of study-specific response evaluation criteria, and prior studies have shown high interobserver variability. 31 In addition, there is a potential for introduction of evaluation or experimenter bias in the assessment due to patient-provider relationship and knowledge of the treatment arm and clinical course. 32,33 At the University of Michigan Rogel Cancer Center, medical oncologists were previously burdened with response assessment.…”

Section: Introductionmentioning

confidence: 99%

Quantitative Imaging Assessment for Clinical Trials in Oncology

Hersberger¹,

Mendiratta‐Lala

Fischer³

et al. 2019

Journal of the National Comprehensive Cancer Network

View full text Add to dashboard Cite

Background: Objective radiographic assessment is crucial for accurately evaluating therapeutic efficacy and patient outcomes in oncology clinical trials. Imaging assessment workflow can be complex; can vary with institution; may burden medical oncologists, who are often inadequately trained in radiology and response criteria; and can lead to high interobserver variability and investigator bias. This article reviews the development of a tumor response assessment core (TRAC) at a comprehensive cancer center with the goal of providing standardized, objective, unbiased tumor imaging assessments, and highlights the web-based platform and overall workflow. In addition, quantitative response assessments by the medical oncologists, radiologist, and TRAC are compared in a retrospective cohort of patients to determine concordance. Patients and Methods: The TRAC workflow includes an image analyst who pre-reviews scans before review with a board-certified radiologist and then manually uploads annotated data on the proprietary TRAC web portal. Patients previously enrolled in 10 lung cancer clinical trials between January 2005 and December 2015 were identified, and the prospectively collected quantitative response assessments by the medical oncologists were compared with retrospective analysis of the same dataset by a radiologist and TRAC. Results: This study enlisted 49 consecutive patients (53% female) with a median age of 60 years (range, 29–78 years); 2 patients did not meet study criteria and were excluded. A linearly weighted kappa test for concordance for TRAC versus radiologist was substantial at 0.65 (95% CI, 0.46–0.85; standard error [SE], 0.10). The kappa value was moderate at 0.42 (95% CI, 0.20–0.64; SE, 0.11) for TRAC versus oncologists and only fair at 0.34 (95% CI, 0.12–0.55; SE, 0.11) for oncologists versus radiologist. Conclusions: Medical oncologists burdened with the task of tumor measurements in patients on clinical trials may introduce significant variability and investigator bias, with the potential to affect therapeutic response and clinical trial outcomes. Institutional imaging cores may help bridge the gap by providing unbiased and reproducible measurements and enable a leaner workflow.

show abstract

“…criteria used to define them and found that they are influenced by interobserver measurement errors of tumor size. They recommended that waterfall plots should be generated only after central review of imaging by radiologists trained in Response Evaluation Criteria in Solid Tumours 1.1 (RECIST) measurement (31). Waterfall plots depict only the best on-study change in tumor burden relative to baseline for each individual patient and cannot represent the kinetics or dynamics of tumor growth (32).…”

Section: Waterfall Plotsmentioning

confidence: 99%

Current and Evolving Methods to Visualize Biological Data in Cancer Research

Chia

Gedye

Boutros

et al. 2016

JNCI J Natl Cancer Inst

View full text Add to dashboard Cite

Although the measurements of clinical outcomes for cancer treatments have become diverse and complex, there remains a need for clear, easily interpreted representations of patients' experiences. With oncology trials increasingly reporting non-time-to-event outcomes, data visualization has evolved to incorporate parameters such as responses to therapy, duration and degree of response, and novel representations of underlying tumor biology. We review both commonly used and newly developed methods to display outcomes in oncology, with a focus on those that have evolved to represent complex datasets.

show abstract

Use and Misuse of Waterfall Plots

Abstract: Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting results of waterfall plots in the context of clinical trials.

Cited by 18 publications

References 13 publications

Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration's Approval of Oncology Drugs

Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration's Approval of Oncology Drugs

Quantitative Imaging Assessment for Clinical Trials in Oncology

Current and Evolving Methods to Visualize Biological Data in Cancer Research

Contact Info

Product

Resources

About