A mixed-methods exploration of an environment for learning computer programming

Abstract

A mixed-methods approach is evaluated for exploring collaborative behaviour, acceptance and progress surrounding an interactive technology for learning computer programming. A review of literature reveals a compelling case for using mixed-methods approaches when evaluating technology-enhanced-learning environments. Here, ethnographic approaches used for the requirements engineering of computing systems are combined with questionnaire-based feedback and skill tests. These are applied to the 'Ceebot' animated 3D learning environment. Video analysis with workplace observation allowed detailed inspection of problem solving and tacit behaviours. Questionnaires and knowledge tests provided broad sample coverage with insights into subject understanding and overall response to the learning environment. Although relatively low scores in programming tests seemingly contradicted the perception that Ceebot had enhanced understanding of programming, this perception was nevertheless found to be correlated with greater test performance. Video analysis corroborated findings that the learning environment and Ceebot animations were engaging and encouraged constructive collaborative behaviours. Ethnographic observations clearly captured Ceebot's value in providing visual cues for problem-solving discussions and for progress through sharing discoveries. Notably, performance in tests was most highly correlated with greater programming practice (p 50.01). It was apparent that although students had appropriated technology for collaborative working and benefitted from visual and tacit cues provided by Ceebot, they had not necessarily deeply learned the lessons intended. The key value of the 'mixed-methods' approach was that ethnographic observations captured the authenticity of learning behaviours, and thereby strengthened confidence in the interpretation of questionnaire and test findings.

Animating student engagement: The impacts of cartoon instructional videos on learning experience

“…• Three-dimensional still backgrounds (Mather 2015) • 'Concept cartoons' used in science education, referring to illustrative drawings similar to comic strips (Keogh and Naylor 1999;Warwick and Stephenson 2002) • Static graphics with simple movements (Ellis 2004) • Black and white or grayscale illustrations of molecular movements, some with no sound effect (Kelly and Jones 2007;Mayer 1999;Nelson et al 2017) • Avatars (animated characters) that appear in educational videos (Cantley, Prendergast, and Schlindwein 2017;Johnson et al 2013;Lee, Hsiao, and Ho 2014;Rosenberg-Kima et al 2010;Wouters, Paas, and Van Merriënboer 2008).…”

“…Firstly, we combine two major functions of animations: explaining new complex concepts to students (Kelly and Jones 2007;Lowe 2003;Nelson et al 2017) and engaging students' interest with multimedia examples, anecdotes and real-world context (Adam et al 2017;Bassford et al 2016;Mather 2015). Unlike the animations developed by Adam et al (2017), our animations are designed to convey new knowledge to students, who are exposed to these concepts for the first time, rather than to reinforce or apply existing knowledge with (simulated) real-world anecdotes or interactive tasks (Bassford et al 2016;Mather 2015). Further, unlike the black and white animations for science teaching that illustrate molecular movements (e.g.…”

“…Despite the prevalence of cartoon programs, there remains limited research on the use of animations as instructional media in higher education (e.g. Adam et al 2017;Kelly and Jones 2007;Mather 2015;Tasker and Dalton 2008). In the small but growing body of extant literature, animations have been used for two main purposes: (1) explaining basic concepts or (2) providing real-world context and application of existing knowledge.…”

“…For example, Bassford et al (2016) incorporate a realistic, interactive crime scene into science, technology, engineering and mathematics (STEM) courses to engage students in interactive learning. Similarly, in teaching computer programming, Mather (2015) uses an animated virtual landscape in which students interact with robot subjects to complete programming tasks.…”

“…However, this body of research has produced inconclusive and contradictory evidence on how animations affect students' learning outcomes. For example, while some researchers find that animated illustrations or assessments improve student understanding in experimental settings (Lowe 2003;Mather 2015;Sanger, Brecheisen, and Hynek 2001), others show that animated simulations can indeed generate or reinforce misconceptions in science learning (Kelly and Jones 2007;Nelson et al 2017;Lowe 2003). Lowe (2003) concludes that the effectiveness of animations depends largely on their design.…”