This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

The global film industry is shaped by gender inequality. Women are structurally underrepresented in professional roles that include high levels of creative and economic decision-making power, such as directors, writers, and producers. In our study, we ask to what extent the film festival sector, a prestigious sub-field of the film industry, is structured by gender biases. To address this question, we conceptualize the festival sector as a one-mode network consisting of film festivals that are connected through screening the same film, and as a two-mode network consisting of films and festivals. The composition of film core creative teams (incl. directors, writers, and producers) varies by gender. While some films are made by mixed gender core creative teams, others are made by women-only or men-only core creative teams. We use data on 1323 films that circulate among 1523 festivals and apply network analysis to describe and disentangle the structural facets that underlie persistent gender inequality in the film festival sector. In the first step, we analyze to what extent the one-mode film festival network is connected through the circulation of films with varying gender compositions, and what festivals occupy broker positions in this network. In the second step, we analyze to what extent films of different gender compositions are equally distributed across the overall festival landscape. For the first step, we find that the festival network connected through films by women-only core creative teams is much sparser in comparison to the network connected through films by men-only core creative teams. We also find that the majority of festivals obtaining important broker positions in the network, has not signed the 5050 × 2020 Gender Parity Pledge, which has important policy implications. For the second step, we find that films by men-only core creative teams are on average screened at more festivals as compared to films by women-only core creative teams, and that the degree of distribution of films by men-only core creative teams is much more skewed indicating a more pronounced festival hit dynamic.

How can researchers from the creative ML/AI community and sociology of culture engage in fruitful collaboration? How do researchers from both fields think (differently) about creativity and the production of creative work? While the ML community considers creativity as a matter of technical expertise and acumen, social scientists have emphasized the role of embeddedness in cultural production. This perspective aims to bridge both disciplines and proposes a conceptual and methodological toolkit for collaboration. We provide a systematic review of recent research in both fields and offer three perspectives around which to structure interdisciplinary research on cultural production: people, processes, and products. We thereby provide necessary grounding work to support multidisciplinary researchers to navigate conceptual and methodological hurdles in their collaboration. Our research will be of interest to ML researchers and sociologists interested in creativity that aim to conduct innovative research by bridging both fields.

No abstract