Not Only WEIRD but “Uncanny”? A Systematic Review of Diversity in Human–Robot Interaction Research

Abstract

Critical voices within and beyond the scientific community have pointed to a grave matter of concern regarding who is included in research and who is not. Subsequent investigations have revealed an extensive form of sampling bias across a broad range of disciplines that conduct human subjects research called “WEIRD”: Western, Educated, Industrial, Rich, and Democratic. Recent work has indicated that this pattern exists within human–computer interaction (HCI) research, as well. How then does human–robot interaction (HRI) fare? And could there be other patterns of sampling bias at play, perhaps those especially relevant to this field of study? We conducted a systematic review of the premier ACM/IEEE International Conference on Human-Robot Interaction (2006–2022) to discover whether and how WEIRD HRI research is. Importantly, we expanded our purview to other factors of representation highlighted by critical work on inclusion and intersectionality as potentially underreported, overlooked, and even marginalized factors of human diversity. Findings from 827 studies across 749 papers confirm that participants in HRI research also tend to be drawn from WEIRD populations. Moreover, we find evidence of limited, obscured, and possible misrepresentation in participant sampling and reporting along key axes of diversity: sex and gender, race and ethnicity, age, sexuality and family configuration, disability, body type, ideology, and domain expertise. We discuss methodological and ethical implications for recruitment, analysis, and reporting, as well as the significance for HRI as a base of knowledge.

Cultural Bias in Explainable AI Research: A Systematic Analysis

“…We found that non-WEIRD populations were rarely sampled in XAI user research. This finding matches results from studies that explored sampling in HCI and HRI and report that 73-75% of papers tested only WEIRD populations (Linxen et al, 2021;Seaborn et al, 2023). However, our results suggest that the problem may be worse in XAI research, as more than 80% of XAI papers with relevant information involved only WEIRD participants.…”

“…Our findings fill a significant gap in previous studies in HCI and HRI that reported generalizability problems related to WEIRD sampling in these fields (Linxen et al, 2021;Seaborn et al, 2023). This is because these studies did not measure the scope of the generalizations that researchers produced thus leaving it unclear whether methodological shortcomings were involved.…”

“…A related concern is that studies in the behavioral sciences, including the fields of humancomputer interaction (HCI) and human-robot interaction (HRI), found that many researchers predominantly tested only individuals from Western, educated, industrialized, rich, and democratic (WEIRD) countries, even though WEIRD people comprise only 12% of the world population (Rad et al, 2018;Linxen et al, 2021;Seaborn et al, 2023). The field of XAI might have taken countermeasures and be less affected by WEIRD sampling.…”

“…More importantly, while several recent studies report that WEIRD sampling may severely limit the generalizability of HCI or HRI studies (Linxen et al, 2021;Seaborn et al, 2023), these studies do not yet control for the point that studies sampling only individuals from one kind of population can be unproblematic even if there are relevant cultural differences. After all, researchers may tailor their conclusions to their specific sample or study population, making clear that other populations remain to be explored.…”

Interacting with Masculinities: A Scoping Review

“…For instance, do "caring masculinities" look the same in a South Asian WhatsApp group chat compared to a British one? We should be cautious about making assumptions and leaning on generalizations from WEIRD research at CHI [121] and adjacent spaces, such as HRI [234]. Still, HCI is a radical, creative, and political space, welcoming of inclusive knowledge and change in praxis.…”