Questionable Research Practices in Experimental Communication Research: A Systematic Analysis From 1980 to 2013

Questionable and open research practices: attitudes and perceptions among quantitative communication researchers

“…If they find significant results, they may stop data collection at that point (Simmons et al, 2011). Among 239 experimental papers in a selection of communication journals, Matthes et al (2015) find that only four papers reported a priori power analyses. While this does not imply that peeking is common, it does suggest that only a few studies have a pre-specified stopping rule when to halt data collection.…”

“…The studies that did examine the prevalence of QRPs in communication have relied on content analyses of research articles, which, for instance, examine whether the distribution of p-values clusters around .05. These analyses demonstrate that QRPs are fairly widespread (Franco, Malhotra, & Simonovits, 2014;Matthes et al, 2015;Vermeulen et al, 2015). However, content analysis has its limitations: (1) they are based upon the reported analyses, while QRPs are, generally, not reported; (2) they cannot distinguish accidental omissions of details from a scholar's intent to do wrong; and (3) they cannot give information about the perceptions researchers have about the field's research practices.…”

“…Second, researchers accept the premises of the proposed solutions, i.e., ORPs. However, with a few notable exceptions (Elson & Przybylski, 2017;Matthes et al, 2015;Vermeulen et al, 2015), most of the literature on the prevalence of QRPs and acceptability of ORPs are based upon evidence from other disciplines, particularly psychology.…”

Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly

“…The qualitative analysis indicated that 2.9 % of cases involved falsification, 4.4 % involved fabrication, and 4.4 % involved both fabrication and falsification Matthes et al (2015;p. 193) Communication Various QRPs Journal articles There were indications of small and insufficiently justified sample sizes, a lack of reported effect sizes, an indiscriminate removal of cases and items, an increasing inflation of p-values directly below p \ 0.05, and a rising share of verified (as opposed to falsified) hypotheses J Bus Psychol (2016) (2013-to-1990 ratio of the percentage of papers = 10.3) than those between 0.051 and 0.059 (ratio = 3.6).…”

Open Communication Science: A Primer on Why and Some Recommendations for How

“…At present, there is limited information in the literature about the full life cycle of studies in the field of communication (Elson & Przybylski, 2017) as well as in other disciplines. What we typically see is the final product -journal articles in which the bulk of statistical significance tests fall below the magical p < .05 threshold (Vermeulen, Beukeboom, Batenburg, Avramiea, Stoyanov, van de Velde, & Oegema, 2015) with scant information about how exactly it is that we got there (Matthes, Marquart, Naderer, Arendt, Schmuck, & Adam, 2015;Simmons, Nelson, & Simonsohn, 2011). There are numerous reasons for this which I will discuss later; for now, the important thing is that the gaps in insights about how communication science gets produced inhibits our ability to build cumulative knowledge internally within the field (Elson & Przybylski, 2017) and has vast implications for the public's perception of the field and its utility for the world (Przybylski & Weinstein, 2016).…”