Use of ICT and its relationship with performance in examinations: a comparison of the ImpaCT2 project's research findings using pupil‐level, school‐level and multilevel modelling data

Abstract

This paper reports a previously unpublished comparative analysis of data from the ImpaCT2 study investigating the relationship between students' performance in England on national tests and their reported use of information technology (particularly networked technology) for school work, at three age levels (11, 14 and 16), in English, Maths and Science (and in additional subjects at age 16). The new analysis compares three separate analyses of the same data set based on approximately 1100 children drawn from 27 primary and 28 secondary schools: the first analysis considers the individual pupil data and looks particularly for pupil-level (and particularly gender) effects; the second analysis groups the pupils by school, and looks particularly for school-level effects; the third analysis was a more conservative multilevel modelling analysis, carried out by an independent team at the University of Durham, which split the schools into three groups, based on overall high, medium or low level of use of information and communications technology (ICT) in each school. All three analyses reported findings based on mean achievement measured in terms of standardised residual gain scores, where each student's progress was measured against predicted gains calculated from baseline data on reading and Maths ability gathered 2 years earlier. The results indicated broad agreement between each of the three modes of analysis, though effect sizes were slightly smaller in the case of the multilevel modelling. The findings are interpreted in relation to case study data on ICT use in schools gathered as part of the wider ImpaCT2 investigation, but the case is also made that the common rationale for using multilevel modelling with large data sets, namely that whole class effects are more significant determinants of performance than individual effects, is less applicable in this study, because of wide variation in the use by children of ICT for school work at home.

An e-maturity analysis explains intention–behavior disjunctions in technology adoption in UK schools

“…Impact09 was a UK-government funded evaluation of technology use in nine high schools in England that had been selected by Becta (the government agency responsible for encouraging the dissemination and effective use of information and communications technology (ICT) in schools), as representing 'outstanding innovation and academic achievement'. In essence, the report argued that a number of previous studies investigating impact had been limited in that they either focused on a single innovation or reported primarily on institutional level factors (see, for example, Johnson, Cox and Watson, 1994;Harrison et al, 2004). However, such studies paid insufficient attention to the contexts of learning.…”

“…The relatively slow pace of adopting and embedding new technologies in education has been noted and studied for 20 years (Bruce, 1993, Johnson et al, 1994, Cuban, 2001, Harrison et al, 2004.…”

Pedagogy and Learning with ICT

“…The ImpaCT2 evaluation (Harrison et al 2002) showed small but statistically signifi cant gains in Science GCSE examinations for 16-year-olds and in national tests for English at age 11 and science at age 13, as well as positive indications which were not statistically signifi cant in one or two other subjects at all three levels. However, as students reported relatively low levels of use of ICT in lessons (e.g.…”

“…Nevertheless, the ImpaCT2 evaluation of the NGfL found that the most frequent use of computers by students in English, maths and science at school (inside or outside lessons) was by 10-11 year olds, of whom 24 per cent reported frequent use in lessons for English (most weeks or every week). However, more than 50 per cent of students in this age group reported that they used computers in maths and science lessons 'never' or 'hardly ever' and more than 50 per cent of older students (aged 13-14 and 15-16) reported the same very low level of use across all three core subjects in school, both inside and outside lessons (Harrison et al 2002). This compares with Becker's fi ndings from the 1999 Teaching, Learning and Computing Survey in the USA, which showed a very low level of use of technology in the teaching of academic subjects (Becker 2000).…”

“…Nevertheless, its purpose is not to ridicule policy failure but rather to urge a more pro-active role for educational researchers in building scenarios for change and supporting the process of radical rather than incremental reform. (Harrison et al 2002;Somekh et al 2002a)…”