Measuring the Cognitive Load of Learning to Program: A Replication Study

Abstract

Cognitive load (CL) on a learner's working memory has emerged as an influential concept in computing education and beyond. CL is commonly divided in at least two components, intrinsic load (IL) and extraneous load (EL). We seek progress on two questions: (1) How can CL components be measured in the programming domain?(2) How should CL measurement deal with the "third component" of germane load (GL)? We replicate two studies: Morrison and colleagues' [49] evaluation of a questionnaire for self-assessing CL in programming, which is an adaptation of a generic instrument; and Jiang and Kalyuga's [24] study, which found support for a twocomponent measure of CL in language learning, with GL redundant. We crowd-sourced CL data using Morrison's questions at the end of a video tutorial on programming for beginners. A confirmatory factor analysis found strong support for a three-factor model, with factors matching the items intended to capture IL, EL, and GL, respectively. A two-factor model with IL-targeting and GL-targeting items combined gave a poorer fit. Our findings strengthen the claims of discriminant validity and internal reliability for Morrison's CL questionnaire for programming; construct validity for GL remains open, however. We affirm the need for further research on the twocomponent theory of CL and the sensitivity of CL self-assessments to contextual factors. CCS CONCEPTS• Social and professional topics → Computing education.

Should Explanations of Program Code Use Audio, Text, or Both? A Replication Study

“…We confirmed the instrument's internal reliability and discriminant validity by replicating Morrison and colleagues' [67] analyses on our data set (as detailed in [105]).…”

“…Learning. Cognitive load theory tends to assume motivated learners and to view motivation as external to the theory [42,53,96,105]. Load effects such as the modality effect are expected to occur when learners are sufficiently engaged with the materials; that may not have been the case in our study.…”

“…In CER, Morrison and colleagues' version of the instrument has been used in a variety of studies (e.g., [23,36,59,66,92,99]). There is evidence of the instrument's internal reliability and discriminant validity [54,67,105], but doubts remain over its construct validity, especially with respect to the items that target germane load [40,53,55,62,105].…”

“…The cognitive-load questionnaire that we used is more established, but its construct validity is open to question [53,62,105]. Learners may interpret the questions in various ways and may not be able to differentiate between intrinsic and extraneous load.…”

Crowdsourcing in Computing Education Research: Case Amazon MTurk

“…The present article focuses on the latter goal and the methodological lessons we learned. The replications are discussed in more detail in [37,38].…”