Research Worth Using: (Re)Framing Research Evidence Quality for Educational Policymaking and Practice

Abstract

This chapter calls for researchers to reconceptualize research quality from the perspective of its expected use, attending to power dynamics that influence how knowledge is defined, constructed, and validated through the research enterprise. Addressing these concerns when designing and conducting education research can yield more useful research evidence for building more equitable education systems. Anchored in scholarship on research utilization and methodological critiques, the chapter introduces a research quality framework that integrates relevance and rigor through five key dimensions of Research Worth Using: (1) relevance of question: alignment of research topics to practical priorities; (2) theoretical credibility: explanatory strength and coherence of principles investigated; (3) methodological credibility: internal and external credibility of study design and execution; (4) evidentiary credibility: robustness and consistency of cumulative evidence; and (5) relevance of answers: justification for practical application. This framework simultaneously uplifts the voices and needs of policymakers, practitioners, and community members, while elevating standards for excellence in education research. We call attention to the myriad ways in which the quality of evidence generated can be strengthened, before describing implications for curating and using research. We conclude by offering suggestions for applying and further developing the framework.

Democratizing the Development of Evidence

“…For example, programs that attract highly motivated students will tend to be positively associated with outcomes, while programs that involve less well-motivated students will appear to performing poorly. Implying a causal relationship where none exists may direct attention away from levers for change that address underlying causes (Ming & Goldenberg, 2021). Correlational relationships also tend to be substantially larger than causal effects, and failing to distinguish between correlational and causal evidence may result in policymakers dismissing programs with "small" effects that are comparatively large relative to alternatives (Kraft, 2020).…”

“…The application of existing research may be viewed as a form of “wishful extrapolation” (Manski, 2013), particularly if studies are set in contexts dissimilar from their communities. Practitioners and community members can share insights on the extent to which local contextual factors vary from the study context in ways that could affect a proposed program or policy (Ming & Goldenberg, 2021). Ultimately, evidence from current local data may be more convincing than existing research to address questions (Finnigan et al, 2013; Massell et al, 2012; Yohalem & Tseng, 2015).…”

“…Education agencies at the federal, state, and local level attend to evidence derived from standardized tests, yet community stakeholders may view students' sense of safety and belonging at school as valued outcomes. Historically, academic concerns have been privileged over practical concerns (Ming & Goldenberg, 2021), and conventional assumptions tend to center white middle class values, taking a deficit lens to minoritized students (Kirkland, 2019). Navigating differences in what types of evidence stakeholders find relevant and convincing is a challenging but essential aspect of democratizing evidence.…”

“…Conversations with researchers might include discussion of existing data and research related to the challenges and potential solutions to refine the set of questions for the learning agenda. Review of research syntheses can mitigate the potential for an individual study or researcher to have an outsize influence on the discussion (Ming & Goldenberg, 2021). In addition, school staff appreciate efforts to rule out programs that might prove ineffective (Corcoran et al, 2001), and stakeholders may be especially interested in existing evidence if the options under consideration require large commitments of resources (Weiss, 1980).…”

“…First, a diverse group of stakeholders, including communities marginalized by systemic inequalities, should have the opportunity to share viewpoints. Incorporating the perspectives of stakeholders can heighten attention to systemic racism and disenfranchisement of minoritized communities in the process of generating evidence, yielding an evidence base that can inform efforts to improve equity (Ming & Goldenberg, 2021). Second, those leading the process agree to pursue a common interest over individual self-interests, taking into consideration the extent to which actions are expected to benefit the public as opposed to being used for private gain (Hutt & Polikoff, 2020).…”