Impact measurement: towards creating a flexible evaluation design for academic literacy interventions

Abstract

Considering the vast array of academic literacy interventions that are presented both nationally and internationally, and the resources required to present these interventions, it is becoming increasingly important for those who are responsible for these interventions to provide evidence of their impact. The aim of this paper is to provide an overview of instruments that are commonly used to assess impact, and to discuss guidelines regarding the use of these instruments, their strengths and their weaknesses. The instruments are divided into two broad categories, namely those that measure the observable improvement in students' academic literacy abilities between the onset and the completion of an intervention, and those that measure the extent to which these abilities are necessary and applied in students' content subjects. A conceptual evaluation design is then proposed that could be used in evaluating the impact of a range of academic literacy interventions. Avenues to explore in future include testing the design in the South African context. Keywords: academic literacy, programme evaluation, impact measurement, effect, interventions IntroductionDue to a variety of reasons, the foremost of which is possibly inadequate secondary education, the implementation of academic literacy interventions in South African universities has become commonplace (Davies 2010: xi; Cliff 2014: 322; Sebolai 2014: 52). What is still largely lacking, though, is evidence of the effectiveness (or lack thereof) of such interventions, and the Fouché, Van Dyk and Butlerhttp://spil.journals.ac.za 110 extent of the impact they have (Holder, Jones, Robinson and Krass 1999: 20; Carstens and Fletcher 2009a: 319; Storch and Tapper 2009: 218; Terraschke and Wahid 2011: 174; Butler 2013: 80; Sebolai 2014: 52). For the purposes of the current study, impact (or effect) is defined as "1) the observable improvement in academic literacy abilities between the onset and the completion of an academic literacy intervention and 2) the extent to which these abilities are necessary and applied in students' content subjects" (Fouché 2015: 3). The terms "impact" and "effect" are used synonymously for the purposes of the current study.According to Hatry and Newcomer (2010: 678), two reasons for evaluating interventions are firstly to provide accountability to stakeholders in the intervention, and secondly to improve the effectiveness of such programmes. As discussed in Fouché (2015), even though some attempts to assess the impact of academic literacy interventions have been made, the type and variety of research instruments used have rarely been sufficient to validly and reliably determine firstly, whether these interventions have an impact, and secondly, what the degree of this impact might be. Reasons for this include that, typically, too few instruments have been used in most studies conducted thus far to make valid conclusions regarding the impact of modules, that instruments used thus far have been varying to such a degree that it might be diffic...

Impact measurement: quantitatively determining the improvement in students’ academic literacy levels at a South African university

“…Four instruments (discussed below) were identified from the evaluation design proposed in Fouché et al (2016) -see Appendix A for a summary of this evaluation design. These instruments were considered appropriate as they suit the generic nature of the subject (thus, the subject-specific instruments proposed in Fouché et al [2016] would not be appropriate) and were considered feasible, taking into account the wide range of faculties and large number of students serviced by AGLE111. In addition, they could be easily integrated into the subject's existing assessment plan.…”

“…This article investigates whether there was an improvement in students' academic literacy levels between the onset and completion of an academic literacy module at a South African university. This is done by using a combination of instruments selected from a proposed evaluation design for academic literacy interventions, suggested by Fouché, Van Dyk and Butler (2016). A pre-test / post-test design is used, where, firstly, students' results in a validated and reliable generic academic literacy test are considered.…”

“…This article forms part of a larger study in which an evaluation design is proposed which can be used to assess the impact of a variety of academic literacy interventions. Two previous articles have, firstly, considered national and international studies that have aimed to assess the impact of academic literacy interventions in the past (Fouché, 2015), and secondly, proposed a flexible, comprehensive design that could be used to validly and reliably assess the effectiveness of a variety of academic literacy interventions, including discipline-specific interventions, generic interventions, writing centre interventions as well as reading programmes (Fouché, Van Dyk & Butler, 2016). This article uses a variety of instruments proposed in Fouché et al (2016) so as to quantitatively assess the impact of a specific academic literacy intervention at the Potchefstroom Campus of the North-West University.…”