Effects of a data-based decision making intervention on student achievement

Abstract

Data-based decision making (DBDM) is becoming important for teachers due to increasing amounts of digital feedback on student performance. In the quasi-experimental study reported here, teachers, principals, and academic coaches from 42 schools were trained for two years in using the results of half-year interim assessments for providing students with tailor-made instruction. Our results did not show any main effects of this DBDM training trajectory on student achievement but did indicate interaction effects with students' low prior achievement levels and socioeconomic status. Teachers experience difficulties in translating student progress data into adaptive instruction in the classroom. Implications of our findings for teacher professionalization are discussed.

Evidence for the Feedback Role of Performance Measurement Systems

“…Teachers received focused training in using the systems and have a local "data driver" onsite to support their analyses. The dynamic data display, cues to guide analysis of the data, and suggested modes of intervention are all co-variates of success in a meta-analysis of experimental studies that examine the effects of data-based decision making on student performance (Staman et al 2017 describing evidence in a Dutch publication by Faber and Visscher). These results echo Cardinaels and Van Veen-Dirks (2010) who used accounting data in an experimental study of the effect of display format on decision making, finding that graphical displays enhance the decision quality of users who are less proficient with numeric data.…”

“…In spite of policies that encourage or require teachers and schools to use data to improve overall performance, education research on the efficacy of PMSs is limited (Staman et al 2017). Often the research is conducted with small groups of students and teachers, over a short duration, and without an experimental design to address self-selection by students or teachers to be study participants.…”

“…They find evidence of positive effects of the presence of a PMS in some grades for some subjects; however, importantly their data are school-level and do not measure actual use of the PMS by teachers. Staman et al (2017) use experimental design with treatment (42) and control (42) schools, with staff in treatment schools receiving access to and training in how to use interim student assessment data to identify improvement opportunities. They find no main effect on student-level mathematics outcomes of being in a treatment school but do find an interaction effect between treatment and the student having low prior achievement and low socioeconomic status.…”

Evidence for the Feedback Role of Performance Measurement Systems

“…Teachers received focused training in using the systems and have a local "data driver" onsite to support their analyses. The dynamic data display, cues to guide analysis of the data, and suggested modes of intervention are all co-variates of success in a meta-analysis of experimental studies that examine the effects of data-based decision making on student performance (Staman et al 2017 The visual displays use color to highlight performance thresholds. Common reports include textual cues to remind the teacher what to look for and to suggest actions that may be warranted.…”

Tracking the process of data use professional development interventions for instructional improvement: A systematic literature review

“…Although studies on data use PDIs have been conducted in many countries, such as the Focus Intervention in the Netherlands (Staman, Timmermans, & Visscher, 2017), the Learning Schools in New Zealand (Lai, McNaughton, Amituanai-Toloa, Turner, & Hsiao, 2009), and Centre for Data-Driven Reform in Education (CDDRE) intervention in the U.S. (Carlson et al, 2011), few attempts have been made to synthesise the effects of data use PDIs on teacher and student outcomes. For example, Faber and Visscher (2014) conducted a meta-analysis on the use of digital student monitoring systems on student performance and found an average effect size of 0.38.…”