Getting ‘What Works’ working: building blocks for the integration of experimental and improvement science

A new methodological approach for evaluating the impact of educational intervention implementation on learning outcomes

“…Based on narrative direct observations and self-report interviews, a thematic analysis identified four intervention implementation themes; (1) 'teacher support', (2) 'teacher supervision', (3) 'intended implementation' and (4) 'established routine'. Collectively, these themes describe the implementing teachers' experiences (Peterson 2016) and classroom context in which the maths app intervention was situated (Biesta 2010;Humphrey et al 2016). These methods are similar to those used in previous implementation process evaluations (Connolly, Keenan, and Urbanska 2018).…”

“…However, these qualitative themes do not capture the extent to which implementation varied across the 11 participating schools, nor do they elucidate how the extent of variation across these themes might be associated with children's learning outcomes with the maths apps. A key aspect of implementation science involves examining the variability in intervention implementation across contexts and learning from this variability (Peterson 2016). Phase II of this study sought to achieve this through examining the relationship between the four intervention implementation themes and children's learning outcomes with the maths apps.…”

“…As such, a more systematic, mixed-methods approach to evaluating implementation is needed (Oakley et al 2006) that also affords statistical examination of the impact of the implementation process on learning outcomes (Peterson 2016;Shaffer 2011;Thomas 2016). The current study demonstrates a novel and informative methodology for examining intervention implementation within a determinant theoretical framework in the context of a recent RCT that evaluated a maths app intervention implemented across 11 primary schools (Outhwaite et al 2018).…”

Is the educational ‘what works’ agenda working? Critical methodological developments

“…This issue also covers a range of methodological approaches, including systematic reviews and meta-analysis (Green et al 2016;Higgins and Katsipataki 2016), advancements of methods for dealing with the complexity involved in interventions by either improved models (Schweig and Pane 2016;Spybrook, Shi, and Kelcey 2016) and/or integrated approaches (Hanley, Cambers, and Haslam 2016;Peterson 2016).…”

“…In order to alleviate these limitations she then proposes 'What works 2.0' which combines the core elements of experimental and improvement science into a strategy to raise educational achievement with the support of evidence from randomized experiments. 'Central to this combined effort is a focus on identifying and testing mechanisms for improving teaching and learning, as applications of principles from the learning sciences' (Peterson 2016, 1, citing Bransford et al 2000and OECD 2007 (1). Similar ideas are shared by Sardar Anwaruddin who approached this from a different stance in the next paper: while addressing the crisis of representation (i.e.…”