Responding to the growing issue of research reproducibility

Abstract

An increasing number of studies, surveys, and editorials highlight experimental and computational reproducibility and replication issues that appear to pervade most areas of modern science. This perspective examines some of the multiple and complex causes of what has been called a “reproducibility crisis,” which can impact materials, interface/(bio)interphase, and vacuum sciences. Reproducibility issues are not new to science, but they are now appearing in new forms requiring innovative solutions. Drivers include the increasingly multidiscipline, multimethod nature of much advanced science, increased complexity of the problems and systems being addressed, and the large amounts and multiple types of experimental and computational data being collected and analyzed in many studies. Sustained efforts are needed to address the causes of reproducibility problems that can hinder the rate of scientific progress and lower public and political regard for science. The initial efforts of the American Vacuum Society to raise awareness of a new generation of reproducibility challenges and provide tools to help address them serve as examples of mitigating actions that can be undertaken.

Proliferation of Faulty Materials Data Analysis in the Literature

“…Some sub-disciplines in science only require a single analytical/measurement tool or just a few tools for a complete analysis of their systems. In contrast, materials analysis generally requires multiple advanced-characterization techniques to obtain an appropriate understanding of a new thin film or material (Baer & Gilmore, 2018). These techniques typically require an understanding of the physics and chemistry behind them, can be performed in multiple modes, and often require detailed first-principles and/or established empirical/semi-empirical modeling for their data reduction.…”

Does repeat synthesis in materials chemistry obey a power law?

“…Although recommendations for improving repeatability in research have been developed (6)(7)(8)(9), it is not clear if they apply in a simple way to all fields. Before attempting to form recommendations appropriate for materials chemistry, or to decide if they are even needed, it is valuable to gather quantitative data related to experimental replicability.…”

Positive and Negative Contributions of XPS to Reproducibility Issues

“…found that 65% of those responding identified reproducibility as a significant issue [7]. As described by Baer and Gilmore [8],…”

“…Such issues challenge research teams and the review process. Systematic and sustained efforts are needed to address the causes of reproducibilityproblems that can hinder the rate of scientific progress and lower public and political regard for science[8].…”