A Call for Openness in Research Reporting: How to Turn Covert Practices Into Helpful Tools

Abstract

Research articles often give inaccurate information about how researchers developed hypotheses, analyzed data, and drew conclusions. Published articles sometimes report only some of the hypotheses that researchers tested, or some of the statistical analyses that researchers made. Articles often imply that researchers formulated all hypotheses before they examined their data when in fact they added or deleted hypotheses after they made some data analyses.Indeed, such covert practices are so common that new entrants into management research may think they are correct behavior. Yet, these practices create false impressions about the validity of research and they undermine the openness that ought to create trust among researchers.Researchers have tried to halt these practices by labeling them "unethical" but their continued prevalence questions the effectiveness of wholly critical approaches. This article proposes a constructive path toward reform: advocating honesty about actual research processes by adding discussions of inferences drawn after data analyses. Post-hoc data analyses can stimulate important theoretical ideas; running alternative statistical models can deepen understanding of empirical patterns; lack of support for hypotheses can identify incorrect or incomplete theories. The management research culture should encourage these practices. The negative effects result from the lack of explicit reporting about them.

When Does HARKing Hurt? Identifying When Different Types of Undisclosed Post Hoc Hypothesizing Harm Scientific Progress

“…Consistent with this view, a recent survey of over 1,500 researchers found that "selective reporting of results" was regarded as the most important factor contributing to irreproducible research (Baker, 2016). In addition, several commentators have argued that HARKing is one of the questionable research practices that reduces the replicability of published effects (e.g., Aguinis, Cascio, & Ramani, 2017;Hollenbeck & Wright, 2017;John et al, 2012;Kerr, 1998;Mazzola & Deuling, 2013;Munafò et al, 2017;Schwab & Starbuck, 2017;Świątkowski & Dompnier, 2017;Unkelbach, 2016;Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). According to this argument, although many researchers claim to conduct confirmatory tests of a priori hypotheses, they actually conduct multiple exploratory tests that are uncorrected for having a greater chance of detecting false positive results.…”

“…The second approach is to change the culture of the scientific community so that it is more accepting of exploratory research (e.g., Hollenbeck & Wright, 2017;Schwab & Starbuck, 2017). Such a change is intended to reduce researchers' motivation to engage in undisclosed HARKing.…”

“…Second, it can produce complex hypotheses that contain many caveats and qualifications in order to account for complex and irregular patterns of results (e.g., prejudice only increases self-esteem among black women who are aged 50 years or more). Hitchcock and Sober (2004) described this process as overfitting because the resulting hypotheses accommodate not only general effects that exist in the population, but also nonreplicable idiosyncratic effects that are limited to the specific sample (see also Bosco et al, 2016;Gigerenzer, 2004;Schwab & Starbuck, 2017). Overfitting is problematic because it increases the probability that hypotheses will be falsified in future samples.…”

“…Suppressing hypotheses after the results are known, or SHARKing, can be harmful to science because it precludes the reported falsification of hypotheses. When associated results are also suppressed, SHARKing can bias information about the size and replicability of effects in meta-analyses (Bosco et al, 2016;Schwab & Starbuck, 2017). Nonetheless, there are some conditions under which SHARKing may not impede scientific progress.…”

“…He also differentiated between different types of HARKing and argued that they are unlikely to be equivalent in terms of their potential costs to scientific progress. However, neither he nor subsequent discussants have explored this issue any further (Hollenbeck & Wright, 2017;Kerr, 1998;Leung, 2011;Schwab & Starbuck, 2017). In particular, it remains unclear when different types of HARKing will be most likely and least likely to harm science.…”