Quantifying Replicability and Consistency in Systematic Reviews

Jaljuli, Iman; Benjamini, Yoav; Shenhav, Liat; Panagiotou, Orestis A.; Heller, Ruth

doi:10.1080/19466315.2022.2050291

Cited by 8 publications

(8 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…). This debate has brought about the development of formal statistical methods for assessing replicability (Benjamini and Heller 2008;Friston, Penny, and Glaser 2005;Jaljuli et al 2023;Wang et al 2022). Replicability analysis aims to identify the findings that are replicated across independent studies that examine the same features.…”

Section: Conflicts Of Interestmentioning

confidence: 99%

Combining Partial True Discovery Guarantee Procedures

Xu,

Solari,

Goeman

2024

Biometrical J

View full text Add to dashboard Cite

Closed testing has recently been shown to be optimal for simultaneous true discovery proportion control. It is, however, challenging to construct true discovery guarantee procedures in such a way that it focuses power on some feature sets chosen by users based on their specific interest or expertise. We propose a procedure that allows users to target power on prespecified feature sets, that is, “focus sets.” Still, the method also allows inference for feature sets chosen post hoc, that is, “nonfocus sets,” for which we deduce a true discovery lower confidence bound by interpolation. Our procedure is built from partial true discovery guarantee procedures combined with Holm's procedure and is a conservative shortcut to the closed testing procedure. A simulation study confirms that the statistical power of our method is relatively high for focus sets, at the cost of power for nonfocus sets, as desired. In addition, we investigate its power property for sets with specific structures, for example, trees and directed acyclic graphs. We also compare our method with AdaFilter in the context of replicability analysis. The application of our method is illustrated with a gene ontology analysis in gene expression data.

show abstract

Section: Conflicts Of Interestmentioning

confidence: 99%

Combining Partial True Discovery Guarantee Procedures

Xu,

Solari,

Goeman

2024

Biometrical J

View full text Add to dashboard Cite

show abstract

“…Independent replication is critical for increasing the discovery power and assessing and establishing replicability of findings across multiple studies through evidence synthesis, thereby providing reassurances about the credibility of results from studies using RCHD. 10…”

Section: Need For Replicationmentioning

confidence: 99%

“…The optimal balance requires subjective judgments and considerations of monetary and nonmonetary costs, which vary across fields and applications. Independent replication is critical for increasing the discovery power and assessing and establishing replicability of findings across multiple studies through evidence synthesis, thereby providing reassurances about the credibility of results from studies using RCHD …”

Section: Need For Replicationmentioning

confidence: 99%

Inferential Challenges for Real-world Evidence in the Era of Routinely Collected Health Data

Panagiotou

Heller

2021

JAMA Oncol

View full text Add to dashboard Cite

Routinely collected health data (RCHD) from administrative claims, electronic medical records, disease registries, and other sources transform the way in which knowledge about cancer is generated. Created and collected as part of routine oncological practice, these data capture cancer-related care provided to real-world patients in real-world settings. As with other types of realworld data, including data generated by patients through websites and wearable devices, measures of social determinants of health, and environmental exposures, RCHD are increasingly popular among researchers, funders, and policy makers because of the potential to generate actionable evidence. The availability of RCHD at low or no cost for large patient populations with long followed-up periods and almost real-time updates has attracted numerous researchers pursuing a nearly infinite number of research questions using any particular set of RCHD.The overarching goal of RCHD is to improve care and outcomes for patients with cancer, a goal that can only be achieved when analyses provide valid inferences. For years, central to the validity of RCHD studies were issues related to data quality (eg, the ability of RCHD to accurately capture clinical information) and biases originating from the observational nonrandomized nature of these studies (eg, confounding). Thankfully, with advances in informatics and causal inference, there is now an array of approaches to overcome the caveats. However, 2 other distinct properties of RCHD have received little attention despite their potential to undermine the validity of inferences: (1) the use of RCHD for formulating research hypotheses that are then examined with the same data (selective inference), and (2) the pursuing of numerous research questions within the same database (multiple testing).In this Viewpoint, we discuss how biases originating from selective inference and multiple testing can potentially invalidate conclusions from RCHD studies, even when high-quality data are analyzed with state-ofthe-art causal inference methodologies.

show abstract

“…Recently, we have proposed such a standard, known as the directional consistency (DC) criterion, which emphasizes that reproducible effects are expected to show concordant (positive or negative) signs across replication experiments with a high probability [19]. Similar ideas have been successfully adopted in general hypothesis testing, meta-analysis, system reviews, and the statistical analysis of qualitative interactions [40,41,42,43,23]. It is worth noting that the implementation of DC the criterion in [19] is not solely focusing on the variability on a particular direction (i.e., the sign-flipping direction).…”

Section: Modeling and Inference Principlesmentioning

confidence: 99%

“…There is a large volume of existing methodological works from relevant fields (e.g., systematic review and meta-analysis) related to replicability assessment [21,22,23]. Some are motivated by investigating effects of irreproducibility due to specific factors, e.g., publication bias/winner's curse [24,25,26,27,28,15] and batch effects [29,30,31], in specific settings.…”

mentioning

confidence: 99%

Statistical Assessment of Replicability via Bayesian Model Criticism

Zhao¹,

Wen²

2021

Preprint

View full text Add to dashboard Cite

Assessment of replicability is critical to ensure the quality and rigor of scientific research.In this paper, we discuss inference and modeling principles for replicability assessment. Targeting distinct application scenarios, we propose two types of Bayesian model criticism approaches to identify potentially irreproducible results in scientific experiments. They are motivated by established Bayesian prior and posterior predictive model-checking procedures and generalize many existing replicability assessment methods. Finally, we discuss the statistical properties of the proposed replicability assessment approaches and illustrate their usages by simulations and examples of real data analysis, including the data from the Reproducibility Project: Psychology and a systematic review of impacts of pre-existing cardiovascular disease on COVID-19 outcomes.

show abstract

Quantifying Replicability and Consistency in Systematic Reviews

Cited by 8 publications

References 45 publications

Combining Partial True Discovery Guarantee Procedures

Combining Partial True Discovery Guarantee Procedures

Inferential Challenges for Real-world Evidence in the Era of Routinely Collected Health Data

Statistical Assessment of Replicability via Bayesian Model Criticism

Contact Info

Product

Resources

About