Transparency in Measurement Reporting

Abstract

Measuring theoretical concepts, so-called constructs, is a central challenge of Player Experience research. Building on recent work in HCI and psychology, we conducted a systematic literature review to study the transparency of measurement reporting. We accessed the ACM Digital Library to analyze all 48 full papers published at CHI PLAY 2020, of those, 24 papers used self-report measurements and were included in the full review. We assessed specifically, whether researchers reported What, How and Why they measured. We found that researchers matched their measures to the construct under study and that administrative details, such as number of points on a Likert-type scale, were frequently reported. However, definitions of the constructs to be measured and justifications for selecting a particular scale were sparse. Lack of transparency in these areas threaten the validity of singular studies, but further compromise the building of theories and accumulation of research knowledge in meta-analytic work. This work is limited to only assessing the current transparency of measurement reporting at CHI PLAY 2020, however we argue this constitutes a fair foundation to assess potential pitfalls. To address these pitfalls, we propose a prescriptive model of a measurement selection process, which aids researchers to systematically define their constructs, specify operationalizations, and justify why these measures were chosen. Future research employing this model should contribute to more transparency in measurement reporting. The research was funded through internal resources. All materials are available on https://osf.io/4xz2v/.

Systematic review on the methodological rigor and effectiveness of digital game-based learning in education

“…Further transparency difficulties were found with the validation process of the outcome measurement. Transparency concerning measurement reporting has become an increasingly important topic in PX [28]. However, many studies still need to report any process of validating their measurement.…”

“…Measurement validation is necessary to give researchers confidence in the findings resulting from their measures. However, in player experience research, questionnaires that have yet to provide evidence of validity are often used LEARNING IN EDUCATION -MAY 3, 2024 [28]. Therefore, when analyzing the quality of intervention studies, it is necessary to investigate whether or not the measurements used for the study outcomes are appropriate.…”

“…Further, consistent with the review of [53], one source of uncertainty is the imprecise and different conceptualizations of engagement used in the studies. A lack of clearly defined constructs is also a general issue within PX research [28]. Future research should clearly define their dependent variables, such as motivation, enjoyment, or engagement, and empirically investigate the motivating character of educational games.…”

Independent Validation of the Player Experience Inventory: Findings from a Large Set of Video Game Players

“…Furthermore, these consistent wordings can also lead to complex sub-factors, such as immersion, being limited in the breadth with which they construct this experience. Statistical validation can not account for content validation, and therefore, we fundamentally can not determine whether the items presented in the PXI genuinely reflect the experiences they wish to measure [2]. These challenges to construct validity are as old as the method itself [4].…”

“…Given that the quality of research findings depends on the reliability and validity of the methods used [2], it is of utmost importance for researchers studying digital games to have an adequately validated scale for measuring players' experiences. The PXI is a promising scale, with its focus on both the psychosocial consequences of gameplay and the functional consequences of game mechanics.…”