Will We March to Utopia, or Be Dragged There? Past Failures and Future Hopes for Publishing Our Science

Giner‐Sorolla, Roger

doi:10.1080/1047840x.2012.706506

Cited by 4 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, as we already pointed out, even a small multiverse analysis is superior to a single-universe analysis in terms of transparency [87]. Thus for researchers who want to signal or promote transparency the option of writing emars can still be attractive, and may become more and more attractive as signals of research integrity and transparency get more and more rewarded [42,43].…”

Section: Discussionmentioning

confidence: 90%

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Dragicevic

Jansen

Sarma

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

113

110

View full text Add to dashboard Cite

We present explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself. This approach draws from two recent ideas: i) multiverse analysis, a philosophy of statistical reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are; and ii) explorable explanations, narratives that can be read as normal explanations but where the reader can also become active by dynamically changing some elements of the explanation. Based on five examples and a design space analysis, we show how combining those two ideas can complement existing reporting approaches and constitute a step towards more transparent research papers. CCS CONCEPTS • Human-centered computing → Human computer interaction (HCI).

show abstract

Section: Discussionmentioning

confidence: 90%

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Dragicevic

Jansen

Sarma

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

113

110

View full text Add to dashboard Cite

show abstract

“…Previous research has outlined the problems with the current publishing system and made suggestions about how to improve the system (Gottfredson, 1978 ; Rosenthal, 1979 ; Ioannidis, 2005 , 2012a , b ; Benos et al, 2007 ; Björk, 2007 ; Birukou et al, 2011 ; Simmons et al, 2011 ; Bekkers, 2012 ; Giner-Sorolla, 2012 ; John et al, 2012 ; Kriegeskorte et al, 2012 ; Lee, 2012 ; Nosek and Bar-Anan, 2012 ; Asendorpf et al, 2013 ). Instead of outlining this work, here I examine how the scientific literature (especially Psychology) can help us understand the problems and develop/implement more effective solutions.…”

mentioning

confidence: 99%

Using science and psychology to improve the dissemination and evaluation of scientific work

Buttliere¹

2014

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Here I outline some of what science can tell us about the problems in psychological publishing and how to best address those problems. First, the motivation behind questionable research practices is examined (the desire to get ahead or, at least, not fall behind). Next, behavior modification strategies are discussed, pointing out that reward works better than punishment. Humans are utility seekers and the implementation of current change initiatives is hindered by high initial buy-in costs and insufficient expected utility. Open science tools interested in improving science should team up, to increase utility while lowering the cost and risk associated with engagement. The best way to realign individual and group motives will probably be to create one, centralized, easy to use, platform, with a profile, a feed of targeted science stories based upon previous system interaction, a sophisticated (public) discussion section, and impact metrics which use the associated data. These measures encourage high quality review and other prosocial activities while inhibiting self-serving behavior. Some advantages of centrally digitizing communications are outlined, including ways the data could be used to improve the peer review process. Most generally, it seems that decisions about change design and implementation should be theory and data driven.

show abstract

“…Aside from examining questions directly related to the most recent debate, we were also interested in gathering opinions on what psychologists think should be done to help move the field forward more generally. Thus, here we asked respondents about 10 of the most discussed proposals for improving the reproducibility and replicability of findings (e.g., Nosek & Lakens, 2014;Asendorpf et al, 2013;Simmons, Nelson, & Simonshon, 2011;Giner-Sorolla, 2012;Levelt committee, 2012;Buttliere, 2014;Kriegskorte, 2012).…”

Section: What Is the Future Of Psychology?mentioning

confidence: 99%

Opinions on the value of direct replication: A survey of 2,000 psychologists

Buttliere¹,

Wicherts²

2018

Preprint

View full text Add to dashboard Cite

A major debatehas emerged in Psychology on the merits of direct replication, with much discussion but little empirical data. Here we conducted a survey among 2,056 psychologists to gather opinions on several important themes in the debate such as: whether direct replications are valuable, whether obtained results can be the outcome of context or expertise, the intentions and reputational interests of the researchers involved, the severity of methodological problems, and what should be done to help move the field forward. Descriptively, the results of our survey suggest that most psychologists trust the seminal findings in the field and are sensitive to the contextual variables which may lead to a failed replication of such a finding; while also agreeing that more direct replications should be done. Also of interest was what opinions were related to valuing direct replications. The application of structural equation models, developed on a subsample then confirmed across subfields, suggests that the respondents’ perceptions of the severity of the methodological problems positively predicts support for direct replication, while respondents’ view on replicator’s intentions (e.g., replication bullying) negatively predicts support for direct replication.

show abstract

Will We March to Utopia, or Be Dragged There? Past Failures and Future Hopes for Publishing Our Science

Cited by 4 publications

References 12 publications

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Using science and psychology to improve the dissemination and evaluation of scientific work

Opinions on the value of direct replication: A survey of 2,000 psychologists

Contact Info

Product

Resources

About