A new methodological approach for evaluating the impact of educational intervention implementation on learning outcomes

Outhwaite, Laura A.; Gulliford, Anthea; Pitchford, Nicola

doi:10.1080/1743727x.2019.1657081

Cited by 31 publications

(23 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One of the most pressing needs is to evaluate the impact of magic-based interventions by comparing their effect to that of a control group or another pedagogical intervention. There exists a large literature on how to develop and test such interventions within educational settings (see, e.g., Wagner, 1997;Outhwaite, Gulliford & Pitchford, 2019), and many of the debates and designs within that literature are applicable to the work reviewed in this paper. Second, future researchers might also consider employing ecologically valid variables, including those relating to knowledge retention (e.g., exam results and test scores), attention and engagement (e.g., class attendance, question-asking and eye tracking), attitudinal shifts (e.g., beliefs about science, pseudoscience and the paranormal) and emotional impact (e.g., validated mood questionnaires).…”

Section: Discussionmentioning

confidence: 99%

Conjuring cognition: a review of educational magic-based interventions

Wiseman

Watt

2020

PeerJ

View full text Add to dashboard Cite

For hundreds of years, magic tricks have been employed within a variety of pedagogic contexts, including promoting science and mathematics, delivering educational messaging, enhancing scepticism about the paranormal, and boosting creative thinking for product design. This review examines this diverse body of work, focusing on studies that have assessed the impact of such interventions. Although the studies tended to yield positive outcomes, much of the work suffered from methodological shortcomings, including measuring the impact of interventions over a relatively short period of time, focusing on self-report measures and failing to employ control groups. The paper makes several recommendations for future study in the area, including assessing the longer-term impact of magic-based interventions, comparing these interventions to other types of pedagogic techniques, focussing on knowledge retention and behavioural outcomes, and collaborating with magicians to develop more impactful interventions.

show abstract

Section: Discussionmentioning

confidence: 99%

Conjuring cognition: a review of educational magic-based interventions

Wiseman

Watt

2020

PeerJ

View full text Add to dashboard Cite

show abstract

“…The fidelity instrument required coders to record concrete, observable behaviors reflective of curriculum implementation. Codes were designed to capture both dosage (i.e., how much time of the day was spent delivering the curriculum) and adherence (i.e., whether the structure and sequence of curriculum activities were followed, see Outhwaite et al, 2019). Because codes explicitly tapped the complexity of the curriculum, it was necessary for coders to be highly familiar with the curriculum itself.…”

Section: Methodsmentioning

confidence: 99%

Effects of Prekindergarten Curricula: Tools of the Mind as a Case Study

Nesbitt

Farran

2021

Monographs Society Res Child

View full text Add to dashboard Cite

Research demonstrates that children's participation in quality early childhood care and education often has immediate positive effects on their social‐emotional, self‐regulation, and achievement outcomes. Most of the research on the impacts of early child care and education has focused narrowly on the United States, but advocacy for economic and social investment in early childhood care and education to support future children's growth and well‐being now exists on an international scale. The longer‐term outcomes from prekindergarten programs have not been as strong. To improve children's long‐term outcomes, one suggested strategy is an intentional, scripted curriculum. Our goal in this monograph is to provide a fully integrated and comprehensive account of a large‐scale, longitudinal, field‐based randomized control trial of the Tools of the Mind (Internal consistency of the Tools) prekindergarten curriculum that occurred in the United States. Our intent is twofold. First, we examine the impact of the Tools curriculum itself, addressing both the potential impacts of the curriculum to improve prekindergarten quality and children's academic, executive function, self‐regulation, and social outcomes. Second, we consider the broader question of whether the use of intentional, scripted curricula during early education can, more generally, enhance both short‐ and long‐term outcomes in children. Developed from a Vygotskian framework, Tools focuses on equipping children with cognitive tools for learning that they can then apply to the task of acquiring and sustaining academic knowledge as well as behavioral competencies. Thus, Tools is an integrated, comprehensive curriculum, not a supplementary one. The Tools approach follows from a socio‐cultural perspective on child development that emphasizes children's acquisition of skills and cultural tools in collaboration with knowledgeable others. The methodology of the 4‐year longitudinal cluster randomized control trial is described in detail. We provide comprehensive information about recruitment, randomization of treatment condition, child assessment instrumentation and procedures, as well as observational assessments, including fidelity of implementation and teacher and child classroom behaviors. We provide results comparing 32 classrooms assigned to the Tools condition and 28 assigned to the business‐as‐usual control condition for children's academic, executive function, self‐regulation, and social gains from prekindergarten to the end of first grade. Developers of the curriculum specifically expected to see benefits on these measures. There were no positive effects for Tools on any of the outcomes. The lack of expected curriculum effects required careful consideration and raised more general questions about how curriculum experiences manifest themselves in assessed skills. As a first step to understanding the findings, we focused on teachers who were implementing Tools and examined the degree to which the curriculum was delivered as intended and the relations betw...

show abstract

“…Interview guides for school staff and study facilitators were derived from the logic model, CFIR online resources (e.g., https://c rguide.org/evaluation-design/qualitative-data/), and the broader literature investigating the delivery of interventions in school settings (44,45; see Additional File 1 for interview guides). Interviews will provide information about both implementation predictors and outcomes.…”

Section: Individual Interviewsmentioning

confidence: 99%

Protocol for the Process Evaluation of a Complex Intervention Delivered in Schools to Prevent Adolescent Depression: The Future Proofing Study

Beames

Lingam

Boydell

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Process evaluations provide insight into how interventions are delivered across varying contexts and why interventions work in some contexts and not in others. This manuscript outlines the detailed protocol for a process evaluation embedded in a cluster randomised trial of a digital depression prevention intervention delivered to secondary school students (The Future Proofing Study). The purpose of this manuscript is to describe the methods that will be used to capture process evaluation data within this trial. Methods: A mixed methods design will be used for this process evaluation with data collected in the intervention arm of the Future Proofing Study. Data collection methods will include semi-structured interviews with school staff members and study facilitators, automatically collected intervention usage and completion data, and participant questionnaires (completed by school staff, school counsellors, study facilitators, and school students). Information will be collected about: i) how the intervention was implemented in schools, including fidelity; ii) school contextual factors and their association with intervention reach (including completion), uptake and acceptability; iii) how school staff, study facilitators, and students and responded to delivering or completing the intervention. How these factors relate to trial effectiveness outcomes will also be assessed. Overall synthesis of the data will provide school cluster-level and individual-level process outcomes. Discussion: This process evaluation provides a detailed description of how to embed a process evaluation into a trial of a complex intervention in schools. It will provide important information about the Future Proofing Study from the perspectives of key groups necessary for implementation – school-teachers, school counsellors and other school staff, as well as study facilitators and students. An understanding of school contextual factors that hinder or facilitate intervention reach and uptake will be provided. Results will also contextualise the trial findings with respect to how the intervention may have worked in some schools but not in others. Furthermore, this process evaluation will inform the development of a model for rolling out digital interventions for the prevention of mental illness across the school sector. Trial Registration: The randomised controlled trial has been prospectively registered: ACTRN12619000855123. Registered 31 May 2019, https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377664&isReview=trueContributions to the Literature· Process evaluations are critical for understanding how interventions are delivered in real world contexts but are rarely embedded in randomised controlled trials or detailed in prospective protocols. · This protocol is the first to illustrate how to embed a process evaluation into a school-based trial of a digital depression prevention intervention, including the use of flexible and pragmatic quantitative and qualitative data collection methods with key stakeholders. · School context factors will be rigorously assessed and analysed for contribution to intervention reach and completion, uptake of study processes, and clinical outcomes. · Study outcomes will guide sustainability efforts in the delivery of digital depression prevention interventions in schools.

show abstract

A new methodological approach for evaluating the impact of educational intervention implementation on learning outcomes

Cited by 31 publications

References 55 publications

Conjuring cognition: a review of educational magic-based interventions

Conjuring cognition: a review of educational magic-based interventions

Effects of Prekindergarten Curricula: Tools of the Mind as a Case Study

Protocol for the Process Evaluation of a Complex Intervention Delivered in Schools to Prevent Adolescent Depression: The Future Proofing Study

Contact Info

Product

Resources

About