Is the Political Slant of Psychology Research Related to Scientific Replicability?

Abstract

Social science researchers are predominantly liberal, and critics have argued this representation may reduce the robustness of research by embedding liberal values into the research process. In an adversarial collaboration, we examined whether the political slant of research findings in psychology is associated with lower rates of scientific replicability. We analyzed 194 original psychology articles reporting studies that had been subject to a later replication attempt ( N = 1,331,413 participants across replications) by having psychology doctoral students (Study 1) and an online sample of U.S. residents (Study 2) from across the political spectrum code the political slant (liberal vs. conservative) of the original research abstracts. The methods and analyses were preregistered. In both studies, the liberal or conservative slant of the original research was not associated with whether the results were successfully replicated. The results remained consistent regardless of the ideology of the coder. Political slant was unrelated to both subsequent citation patterns and the original study’s effect size and not consistently related to the original study’s sample size. However, we found modest evidence that research with greater political slant—whether liberal or conservative—was less replicable, whereas statistical robustness consistently predicted replication success. We discuss the implications for social science, politics, and replicability.

Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences

“…To address this third question, we used an openly available dataset that included data from eight different replication projects, including the Reproducibility project [ 58 ], Many Labs 1 [ 59 ], Many Labs 2 [ 56 ] and Many Labs 3 [ 55 ], a special issue of Social Psychology [ 60 ], the Association of Psychological Sciences Registered Reports Repository [ 61 ], the pre-publication independent replication repository [ 62 ] and Curate Science [ 63 ]. Reinero et al [ 40 ] collated these data, which also includes the number of citations to each original article, the number of authors on the original article, year of publication and the Altmetric score for the original article. The full dataset curated by Reinero et al [ 40 ] is available on the Open Science Framework ( ).…”

“…Reinero et al [ 40 ] collated these data, which also includes the number of citations to each original article, the number of authors on the original article, year of publication and the Altmetric score for the original article. The full dataset curated by Reinero et al [ 40 ] is available on the Open Science Framework ( ). We merged these data with the 2017 JIFs obtained using the scholar [ 50 ] package in R and coded each journal by subdiscipline (clinical, cognitive, social, judgement and decision making, general, marketing and ‘other’).…”

“…The analyses reported herein bring together three data sources to address whether citation counts and JIFs have as much utility for assessing research quality as how they are used in practice. These data sources include a dataset used in a large-scale assessment of statistical reporting inconsistencies across 50 845 articles in behavioural science and neuroscience journals published between 1985 and 2016 [ 39 ] and 190 replication studies from a collection of repositories (see [ 40 ]). Each dataset was merged with the 2017 JIFs and/or article-level citation counts.…”

Trends in American scientists’ political donations and implications for trust in science

“…Claims that scientists distort research for the sake of liberal political agendas have not been substantiated by any solid evidence (Larregue, 2018 ). For instance, one study looking at political slant in psychology research (a field largely dominated by liberal scholars) found that findings of research articles whose abstracts were coded as liberal (i.e., articles whose research conclusions were more consistent with a liberal worldview) were just as likely to be replicable and as statistically robust as the findings of research articles whose abstracts were coded as having a conservative slant (Reinero et al, 2020 ).…”

A Replication of Stern, West, and Schmitt (2014) Indicates Less False Consensus Among Liberals Than Conservatives, But No False Uniqueness

“…A liberal illusion of uniqueness is also relevant to ongoing debates over the presence or scale of left-right asymmetries and the replicability of political psychology (Reinero et al, 2020). For instance, Ditto and colleagues (2019) conducted meta-analyses illustrating balance between liberals’ and conservatives’ partisan biases.…”