Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results

Abstract

The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.

Questionable and open research practices: attitudes and perceptions among quantitative communication researchers

“…A failure to disclose all outcome variables seems relatively common in the social sciences. Franco et al (2015Franco et al ( , 2016 found that 60% (N=53) of the experimental political science papers and 70% (N=32) of experimental psychology papers they assessed reported fewer outcome variables in the published study compared to the number of outcome variables that were actually included in the study.…”

“…The primary way communication researchers have examined the prevalence of QRPs is through content analysis (Matthes et al, 2015;Vermeulen et al, 2015). This is certainly a valid approach for identifying certain QRPs such as publication bias, non-publication of studies or outcome variables, and rounding (Franco et al, 2014(Franco et al, , 2015Hartgerink et al, 2016). But other QRPs, such as imputing missing data, peeking at data, running additional analyses, are harder to isolate with a content analysis of the published literature as these QRPs are generally not disclosed in the paper.…”

Transparency of CHI Research Artifacts: Results of a Self-Reported Survey

“…Recently, scientific disciplines such as psychology [9], medicine [3], economics [6], and political science [16] have had many published research articles fail to replicate, i.e. rerunning the experiment with more statistical power does not yield an effect as strong as or in the same direction as the original experiment.…”

Forschung als soziales Dilemma: Eine meta-wissenschaftliche Bestandsaufnahme zur Glaubwürdigkeit politikwissenschaftlicher Befunde und ein Appell zur Veränderung akademischer Anreizstrukturen

“…Examining the race discrimination literature, Zigerell (2017) demonstrated several cases of selective reporting in which alternative model specifications remained unreported when they led to different results. In a large-scale examination of 249 political science studies from a research competition for which data and research questions were known, Franco et al (2015) showed that only one out of five published studies reported all experimental conditions and outcome variables. The average study left 0.5 experimental conditions and 3.1 experimental outcomes undisclosed.…”