Investigating the Impact on Learning Outcomes Through the Use of EdTech During Covid-19: Evidence from an RCT in the Punjab province of Pakistan

Abstract

An output of the EdTech Hub, https://edtechhub.org

Toward a holistic approach to EdTech effectiveness: Lessons from Covid-19 research in Bangladesh, Ghana, Kenya, Pakistan, and Sierra Leone

“…The issue of how to measure the impact on learning outcomes was a primary concern for nearly all of our studies, with most of them measuring learning gains directly ( Adil et al, 2021 , Afoakwah et al, 2021 , Amenya et al, 2021 , Aurino et al, 2022 , Islam et al, 2022 ), with others also collecting data in relation to perceived impacts on learning outcomes reported by participants (e.g., Hodor et al, 2021 ). Assessment tools used were mainly focused on foundational learning outcomes, often using instruments which have been developed and validated independently and composed of test items for both literacy and numeracy.…”

“…It is notable that the availability of existing validated tools, which could be rapidly deployed, allowed the collection of data on learning gains despite the relatively short timescale and crisis context. Student learning assessment tools used by the studies included: Adil et al (2021) : Language and mathematics tests administered for the ASER (Annual Status of Education Report) Survey, and grade-appropriate tests based on the Punjab textbook board syllabus for maths, English, and Urdu for older children. Afoakwah et al (2021) : Grade 4 TIMSS (Trends in International Mathematics and Science Study) items ( TIMSS, 2013 ); Amenya et al (2021) : SeGRA (Senior Grade Reading Assessment) and SeGMA (Senior Grade Mathematics Assessment) covering literacy and mathematics for older pupils; Aurino et al (2022) : Literacy and numeracy were both assessed, using items from the IDELA (International Development and Early Learning Assessment; Pisani et al, 2018 ), EGRA (Early Grade Reading Assessment; RTI International, 2016 ), EGMA (Early Grade Math Assessment; Platas et al, 2014 ) and the Young Lives surveys ( Barnett et al, 2013 ); Islam et al, 2022 : Assessment taken from the national curriculum of Bangladesh covering English literacy, Bangla literacy, numeracy, and general knowledge ( Islam et al, 2022 , p.60); plus, Renzulli scales to measure noncognitive outcomes (with mothers as respondents) ( Renzulli et al, 2002 ).…”

“… Adil et al (2021) : Language and mathematics tests administered for the ASER (Annual Status of Education Report) Survey, and grade-appropriate tests based on the Punjab textbook board syllabus for maths, English, and Urdu for older children.…”

“…However, positive effects were less pronounced in another IVR study in Ghana, only impacting one aspect of numeracy: place value knowledge ( Afoakwah et al, 2021 ). Online TaRL (in Pakistan) had variable effects for Grade 8 students, causing language scores to go up but not affecting maths scores ( Adil et al, 2021 ).…”

“…Furthermore, beyond the domains of literacy and numeracy, the range of types of learning outcomes for which robust tools are available is limited ( Anderson, 2019 ). An observation which emerged from the analysis was that when used, assessment was found to contribute to improved learning, but there was also concern about a lack of transparency about what would be assessed and how it would be used ( Adil et al, 2021 ), which is a key practical consideration for future research designs.…”