Enhancing Credibility of Chemical Safety Studies: Emerging Consensus on Key Assessment Criteria

Abstract

ObjectivesWe examined the extent to which consensus exists on the criteria that should be used for assessing the credibility of a scientific work, regardless of its funding source, and explored how these criteria might be implemented.Data sourcesThree publications, all presented at a session of the 2009 annual meeting of the Society for Risk Analysis, have proposed a range of criteria for evaluating the credibility of scientific studies. At least two other similar sets of criteria have recently been proposed elsewhere.Data extraction/synthesisIn this article we review these criteria, highlight the commonalities among them, and integrate them into a list of 10 criteria. We also discuss issues inherent in any attempt to implement the criteria systematically.ConclusionsRecommendations by many scientists and policy experts converge on a finite list of criteria for assessing the credibility of a scientific study without regard to funding source. These criteria should be formalized through a consensus process or a governmental initiative that includes discussion and pilot application of a system for reproducibly implementing them. Formal establishment of such a system should enable the debate regarding chemical studies to move beyond funding issues and focus on scientific merit.

Evidence-based toxicology for the 21st century: Opportunities and challenges

“…the launching of the eBtC is timely, as the toxicology literature increasingly invokes eB-related themes or practices such as transparency in decision-making (Schreider et al, 2010), systematic and transparent reviews of evidence , synthesis of types of evidence to establish causal inference (Adami et al, 2011), and assessment of bias/credibility (Conrad and Becker, 2010). We also see practical examples of the application of evidence-based methodology or terminology Abhyankar et al, 2011;Maull et al, 2012).…”

The promise and perils of industry‐funded science

“…Whereas academic research has recently come under fire because of such concerns (Begley & Ellis, ; Begley & Ioannidis, ; Open Science Collaboration, ), some researchers contend such problems can be alleviated by working with industry (Edwards, ). In particular, many industry labs follow standardized protocols for performing studies and adhere to Good Laboratory Practice guidelines, which, they argue, include rigorous safeguards for ensuring that those protocols are actually followed (Conrad & Becker, ).…”

“…With respect to the collection and analysis of data, the Good Laboratory Practices (GLP) record‐keeping system may provide an important strategy for preventing outright fabrication and falsification of data (Conrad & Becker, ). However, it is doubtful that GLP and other efforts at standardization are adequate for addressing the more common problems of design bias and rhetorical presentation of study results, and standardization can lock outdated or questionable methods in place (Elliott, ; Wickson & Forsberg, ).…”

Taking Financial Relationships into Account When Assessing Research

“…If reviewers and readers have this information, they can assess the study’s scientific validity and significance. Thus, the critic could insist that, at best, disclosure of financial interests adds nothing of consequence to peer review and scientific debate, and at worst it leads to the unjustifiable dismissal of high-quality research (Borgert, 2007; Conrad and Becker, 2011). If key details of a study are included in the publication, then reviewers and readers should be able to determine whether aims and objectives are well-described, whether relevant variables are measured, whether statistical methods are appropriate, and whether the data support the conclusions.…”

“…In thinking about the impact of financial relationships, one should consider how they should affect a study’s credibility. By “credibility” we mean the justifiability of accepting the results of the study for purposes of future research (such as relying on the results for one’s own research) or for practical or policy implications (such as using information from the study to regulate products or prescribe medications) (see also Conrad and Becker, 2011). Financial relationships could enhance credibility to varying degrees, undermine credibility to varying degrees, or have no impact on credibility.…”