This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

This study introduces and investigates the validity of a brief scale measuring a challenged sense of belonging. The sense of belonging as well as challenges to this sense are important, albeit neglected aspects of social integration and of significance to migration and refugee studies as well as to virtually all other social science contexts. Assessing a challenged or eroded sense of belonging provides important insights into how individuals relate to their environment and whether they feel socially connected or disconnected from it. The construct goes beyond national or cultural identity, instead emphasizing the dynamic processes of emotional attachment. Reviewing the substantial theoretical literature on belonging, we identify four of its key elements: connection, participation, identification, and congruence. Drawing on existing measurement instruments, we propose a brief Challenged Sense of Belonging Scale (CSBS) that addresses each of the four elements and investigate its validity in a unique, multi-lingual random sample of 3783 adult refugees in Germany from various national and cultural backgrounds. We provide evidence for the scale’s validity separately for three main survey languages (English, Arabic, Farsi/Dari) using confirmatory factor analysis, a test of measurement invariance, item test and rest correlations, and correlation analysis to explore convergent validity. Our findings suggest that the scale is a suitable instrument for the assessment of a challenged sense of belonging in a heterogeneous population of refugees.

No abstract

No abstract