Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

Abstract

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives have emphasized the benefits of several transparency and reproducibility-related research practices; however, their adoption across the psychology literature is unknown. To estimate their prevalence, we manually examined a random sample of 250 psychology articles published between 2014-2017. Over half of the articles were publicly available (154/237, 65% [95% confidence interval, 59%-71%]); however, sharing of research materials (26/183, 14% [10%-19%]), study protocols (0/188, 0% [0%-1%]), raw data (4/188, 2% [1%-4%]), and analysis scripts (1/188, 1% [0%-1%]) was rare. Pre-registration was also uncommon (5/188, 3% [1%-5%]). Many articles included a funding disclosure statement (142/228, 62% [56%-69%]), but conflict of interest statements were less common (88/228, 39% [32%-45%]). Replication studies were rare (10/188, 5% [3%-8%]) and few studies were included in systematic reviews (21/183, 11% [8%-16%]) or meta-analyses (12/183, 7% [4%-10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish a baseline which can be used to assess future progress towards increasing the credibility and utility of psychology research.

Correction notices in psychology: impactful or inconsequential?

“…It also follows other meta-scientific studies in psychology that have opted to focus on multiple articles from a small number of journals (typically one to four) rather than randomly selecting articles across an entire domain, thus keeping policies and quality standards constant (e.g. [3,[10][11][12][13]). Figure 2 depicts the corrections-to-articles ratio for the selected journals in the timeframe under consideration.…”

Practice what you preach: Credibility-enhancing displays and the growth of open science

“…We briefly review mechanisms promoting cooperation in social dilemmas , explaining the crucial role of normativity (Bicchieri, 2006). We then argue that the second challenge to spreading OSPs is their non-normativity, summarizing studies which demonstrate that despite growing support for the movement (Tenopir et al, 2015), OSPs remain uncommon (Hardwicke et al, 2020). Taken together, these data suggest a third challenge to spreading OSPs: the temptation to engage in dishonest virtue signaling (Kraft-Todd, Kleiman-Weiner, & Young, 2020).…”

“…Scientists' widespread support of OSPs might suggest that they have expectations that others will engage in OSPs; if so, scientists could accrue personal benefits by fulfilling these expectations. Therefore, scientists may underestimate these benefits because they underestimate others' approval of and engagement in OSPs (Hardwicke et al, 2020). Thus, similar to typical cases CREDs theory considers (e.g.…”

“…Encouragingly, in perhaps the most sophisticated and among the most recent self-report studies of N=2,801 social scientists (21.3% in psychology) in 2018, an estimated upper bound of 73% reported ever having publicly posted their data (although only 51% among psychologists; Christensen et al, 2020). Still, these trends are far from representative across the field as a whole, particularly in cross-sectional analysis; for example, in a contemporary audit study of 250 psychology articles published between 2014 and 2017, only 2% had open data (Hardwicke et al, 2020). These trends are similar for other OSPs: while 60% of psychologists report ever having posted open materials and 44% of psychologists report ever having preregistered an experiment (Christensen et al, 2020), of papers audited, only 14% had open materials and only 3% of papers audited had preregistered experiments (Hardwicke et al, 2020).…”

“…For behavioral scientists who perceive OSPs as having low injunctive normativity-which may be true of those with weak second-order beliefs, i.e. who particularly underestimate their prevalence and support (Hardwicke et al, 2020), as discussed above-they might conceptualize OSPs as (3) novel and non-normative. Reflecting on the incentives behavioral scientists have for engaging in OSPs based on this typology, we can infer that type (1) will be intrinsically motivated to engage in OSPs and types (2) and (3) will be motivated to engage in OSPs to the extent that they perceive stronger expectation from others that they should, while type (3) may also require additional information about others' expectations.…”