Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Abstract

Measuring theoretical concepts, so-called constructs, is a central challenge of Player Experience research. Building on recent work in HCI and psychology, we conducted a systematic literature review to study the transparency of measurement reporting. We accessed the ACM Digital Library to analyze all 48 full papers published at CHI PLAY 2020, of those, 24 papers used self-report measurements and were included in the full review. We assessed specifically, whether researchers reported What, How and Why they measured. We found that researchers matched their measures to the construct under study and that administrative details, such as number of points on a Likert-type scale, were frequently reported. However, definitions of the constructs to be measured and justifications for selecting a particular scale were sparse. Lack of transparency in these areas threaten the validity of singular studies, but further compromise the building of theories and accumulation of research knowledge in meta-analytic work. This work is limited to only assessing the current transparency of measurement reporting at CHI PLAY 2020, however we argue this constitutes a fair foundation to asses potential pitfalls. To address these pitfalls, we propose a prescriptive model of a measurement selection process, which aids researchers to systematically define their constructs, specify operationalizations, and justify why these measures were chosen. Future research employing this model should contribute to more transparency in measurement reporting. The research was funded through internal resources. All materials are available on https://osf.io/4xz2v/.

Modulating Frustration and Agency Using Fabricated Input for Motor Imagery BCIs in Stroke Rehabilitation

“…Hence it might be reasonable to expect how the MI rate only explain a low amount of variance (28% R 2 m ) in users' ratings of perceived control. From an experimental design point of view, the benefits of using surrogate BCI includes allowing for known-groups validation [69] for example by measuring a well-known construct such as 100% control known beforehand to be distinct. The participants, many without prior MI experience, described trying various other approaches when they realized MI did not work consistently, including focusing on the fingertips or clenching their stomach; this could reduce the actual MI recognition rate.…”

Measurement Practices in UX Research: A Systematic Quantitative Literature Review

“…Given that researchers lose many advantages that standardized scales offer (Nunnally, 1978) when instead opting for self-developed items, both of these studies provide an initial indication that UX research might be at general risk of QMPs. Shifting our focus to the field of PX research, a recent literature review by Aeschbach et al (2021) raised concerns regarding a lack of transparency in PX research, which among other things, might lead to QMPs. The study analyzed all 48 full papers published at CHI PLAY 2020, among which 24 used scales for measurement.…”

“…The study analyzed all 48 full papers published at CHI PLAY 2020, among which 24 used scales for measurement. Within this sample, definitions for the constructs investigated were rarely provided, and rationales for scale selection were often lacking (Aeschbach et al, 2021). If a similar lack of transparency is also present in UX research, it might threaten the validity of the field and its findings.…”

“…Recent findings from the field of player experience (PX) research have shown that there is a lack of transparent reporting in studies that employ survey scales for measurement, indicating the presence of QMPs in PX research (Aeschbach et al, 2021).…”

“…However, there is no systematic knowledge of how prevalent QMPs are in UX research. We, therefore, set out to identify MEASUREMENT PRACTICES IN UX RESEARCH: A SQLR 5 current best practices, based on literature from past HCI research, psychometrics, and the social sciences (Aeschbach et al, 2021;DeVellis, 2017;Furr, 2011;Nunnally, 1978), and systematically investigated how well scale-based UX research follows these best practices.…”