Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research

Abstract

Background: As part of a coordinated effort to expand research activity around rigor, reproducibility, and transparency (RRT) across scientific disciplines, a team of investigators at the Indiana University School of Public Health-Bloomington hosted a workshop in October 2019 with international leaders to discuss key opportunities for RRT research. Objective: The workshop aimed to identify research priorities and opportunities related to RRT. Design: Over two-days, workshop attendees gave presentations and participated in three working groups: (1) Improving Education & Training in RRT, (2) Reducing Statistical Errors and Increasing Analytic Transparency, and (3) Looking Outward: Increasing Truthfulness and Accuracy of Research Communications. Following small-group discussions, the working groups presented their findings, and participants discussed the research opportunities identified. The investigators compiled a list of research priorities, which were circulated to all participants for feedback. Results: Participants identified the following priority research questions: (1) Can RRT-focused statistics and mathematical modeling courses improve statistics practice?; (2) Can specialized training in scientific writing improve transparency?; (3) Does modality (e.g. face to face, online) affect the efficacy RRT-related education?; (4) How can automated programs help identify errors more efficiently?; (5) What is the prevalence and impact of errors in scientific publications (e.g., analytic inconsistencies, statistical errors, and other objective errors)?; (6) Do error prevention workflows reduce errors?; (7) How do we encourage post-publication error correction?; (8) How does ‘spin’ in research communication affect stakeholder understanding and use of research evidence?; (9) Do tools to aid writing research reports increase comprehensiveness and clarity of research reports?; and (10) Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting? Conclusion: Participants identified important and relatively unexplored questions related to improving RRT. This list may be useful to the scientific community and investigators seeking to advance meta-science (i.e. research on research).

Optimizing Bug Bounty Programs for Efficient Malware-Related Vulnerability Discovery

“…In the context of mobile security, bug bounty programs play a significant role in addressing vulnerabilities in mobile applications and operating systems, particularly in combating the latest mobile malware, such as mobile banking trojans, cryptocurrency mining, and ransomware ( [14]; [15]). These programs are also likened to "red teams" in scientific research, where methodologists, statisticians, and subject-matter experts critique study designs and analyses, offering incentives akin to bug bounty programs in computer software development [16].…”

Perspective on scientific truth versus scientific evidence; maintaining integrity in global food systems

“…Research is being conducted into why researchers may not be following high RRT standards. Reasons are, for instance, pressure to publish, lack of funding, lack of training in scientific methods, peer-review pressure to acknowledge legacy findings and similar (Anderson et al 2007;Valdez et al 2020). Other researchers systematically outline reasons for why it is difficult to retract and correct published results (Allison et al 2016).…”

An Author’s Guide to Mastering Research Communication Skills: Introduction of a Medical Manuscript

“…A significant amount of the current medical literature -systematic reviews-has a limited capacity to produce credible evidence and generalizable conclusions. This can be attributed to flawed original studies [5] or unjustified generalizations by authors of systematic reviews [6]. And is another indicator of inadequate research communication skills.…”