This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

No abstract

Pre-service teachers tend to devalue general pedagogical knowledge (GPK) as a valid source for deriving successful teaching practices. The present study investigated beliefs about knowledge sources and epistemic beliefs as predictors for students’ perceived value of GPK. Three pre-registered hypotheses were tested. We expected beliefs that GPK originates from scientific sources to entail a devaluation of GPK (Hypothesis 1). Concerning epistemic beliefs, we expected absolute beliefs to positively, and multiplistic beliefs to negatively predict pre-service teachers’ perceived practical value of GPK (Hypothesis 2). Finally, we expected relationships between epistemic beliefs and pre-service teachers’ perceived practical value of GPK to be confounded by epistemic trustworthiness, perceived topic-specific consistency and topic-specific familiarity (Hypothesis 3). In a study using a split plot design, 365 pre-service teachers were presented with four texts on different educational research topics. For each topic, three text versions were constructed. Even though they were invariant in content, these versions varied in a way that the results were allegedly generated by a practitioner, an expert or by means of a scientific study. Unexpectedly, results showed that research findings allegedly generated by means of a scientific study were associated with a higher perceived value of (topic-specific) GPK for practice (Hypothesis 1). As expected, the perceived value of GPK for practice was predicted by topic-specific multiplism and domain-specific absolutism (Hypothesis 2). These predictive effects were confounded by expertise evaluations of the source and the consistency of prior beliefs with the presented research results (Hypothesis 3). In summary, our results suggest that source beliefs might not be responsible for the devaluation of GPK, but that beliefs on the nature and structure of GPK (i.e., epistemic beliefs) might play an even more important role in this respect. Implications for research and practice are discussed.

In-service and preservice teachers are increasingly required to integrate research results into their classroom practice. However, due to their limited methodological background knowledge, they often cannot evaluate scientific evidence firsthand and instead must trust the sources on which they rely. In two experimental studies, we investigated the amount of this so-called epistemic trustworthiness (dimensions expertise, integrity, and benevolence) that student-teachers ascribe to the authors of texts who present classical research findings (e.g., learning with worked-out examples) that allegedly were written by a practitioner, an expert, or a scientist. Results from the first exploratory study suggest that student-teachers view scientists as “smart but evil,” since they rate them as having substantially more expertise than practitioners, while also being less benevolent and lacking in integrity. Moreover, results from the exploratory study suggest that evaluativistic epistemic beliefs (beliefs about the nature of knowledge) predict epistemic trustworthiness. A preregistered conceptual replication study (Study 2) provided more evidence for the “smart but evil” stereotype. Further directions of research as well as implications for practice are discussed.

No abstract