Evaluative reasoning in public-sector evaluation in Aotearoa New Zealand: How are we doing?

Nunns, Heather

doi:10.18296/em.0007

Cited by 12 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The importance of a warranted argument being developed through a chain of evaluative reasoning that connects the evidence to an evaluative conclusion is emphasised by many theorists (Davidson, 2014a;Fournier, 1995;Nunns, Peace, & Witten, 2015;Rog, 1995;Schwandt, 2008). Davidson (2014b, p. 38) lists six steps to evaluative reasoning: listing criteria of merit, determining the relative importance of each, defining levels of performance, gathering and analysing evidence, drawing evaluative conclusions about each of the criteria, then finally, synthesising these into overall evaluative conclusions about the programme as a whole.…”

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

“…Empirical research in Aotearoa New Zealand (Nunns et al, 2015) and Canada (Hurteau, Houle, & Mongiat, 2009) criticises how evaluative conclusions are reached, and the lack of a warranted argument that makes the connection between evidence and evaluative conclusion clear to the reader. Nunns et al (2015) did a meta-analysis of 30 publicly accessible evaluation reports written or commissioned by public-sector agencies during the period 2010-2013 to understand how evaluative reasoning was being practised in Aotearoa New Zealand.…”

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

“…Nunns et al (2015) did a meta-analysis of 30 publicly accessible evaluation reports written or commissioned by public-sector agencies during the period 2010-2013 to understand how evaluative reasoning was being practised in Aotearoa New Zealand. Twenty-four reports contained evaluative conclusions, but only 13 based these on criteria or other comparators defined in the report (Nunns et al, 2015). A similar study was undertaken in Canada, by Hurteau, Houle, and Mongiat (2009).…”

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

“…Julnes (2012a) calls for more empirical evidence to understand the impact of different valuing methodologies. Nunns et al (2015) and Hurteau et al (2009) have sought to understand evaluation practice around warranted evaluative conclusions in Aotearoa New Zealand and Canada respectively. Each reports a snapshot of practice, which, although useful, does not address Julnes' call to understand the implications for evaluation use or validity.…”

Section: Implications For Evaluationmentioning

confidence: 99%

“…Davidson (2014b) identifies that synthesis methodology is not taught in social science research training and Gullickson concludes that there is little-to-no emphasis on synthesis in evaluator teaching or training. Given that this review shows that when synthesis is done it is not always done well (Davidson, 2014b;Hurteau et al, 2009;Nunns et al, 2015), and that literature on different ways to reach an overall evaluative conclusion is sparse, this review adds to the call for this area of evaluation practice to be highlighted in evaluator training and professional development. This could include how to deliberate over findings and conclusions when diverse stakeholders are actively involved (House & Howe, 1999;McDonald, 2008).…”

Section: Implications For Evaluationmentioning

confidence: 99%

See 4 more Smart Citations

The overall evaluative conclusion as an element of evaluation practice

MacDonald¹

2019

View full text Add to dashboard Cite

The questions of who values, with whom, in what ways, and under what conditions concern all evaluators but are explicitly considered by some theorists more than others. Theorists placed on the valuing branch of Christie and Alkin’s (2013) evaluation theory tree emphasise valuing in their conceptualisation of evaluation, but even among these theorists there is diversity in the ways in which valuing is considered and realised in evaluation practice. This article explores this diversity within one aspect of valuing—the valuing involved in reaching a warranted conclusion about the overall merit, worth, or significance of an evaluand. It considers the extent to which the literature discusses overall evaluative conclusions as an element of evaluation practice; the extent to which drawing such conclusions is seen as the responsibility of the evaluator or stakeholders; and the methods that may be used to reach a warranted evaluative conclusion. The author concludes that there has been little empirical research undertaken on the valuing involved in reaching a warranted conclusion about the overall merit, worth, or significance of an evaluand. Much of the literature is evaluators theorising from different epistemological positions. Thus, while the literature does not definitively inform evaluators of whether they should always reach an overall evaluative conclusion, who they should involve, and what methods they should use, this review does support evaluators to reflect on these issues in their practice, and to make deliberate, informed decisions about the making—or not—of overall conclusions or judgements in future evaluations.

show abstract

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

Section: The Dominant Rhetoric: a Criterial Approach That Leads To Symentioning

confidence: 99%

Section: Implications For Evaluationmentioning

confidence: 99%

Section: Implications For Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

The overall evaluative conclusion as an element of evaluation practice

MacDonald¹

2019

View full text Add to dashboard Cite

show abstract

What does implementing case‐centered teaching of evaluation look like in practice?

Kallemeyn

Roseveare

Steinberg

et al. 2021

New Drctns Evaluation

View full text Add to dashboard Cite

Rooted in the pedagogical literature, three evaluation educators, guided by a facilitator, engaged in reflective practice regarding case-centered teaching and learning. We engaged in (D)escription, (A)nalysis, (T)heorizing, and (A)cting (DATA model) in relation to case-centered teaching. Based on a cross-case analysis, we identified five common actions in our teaching with cases: (1) Use case-centered teaching in various contexts to support a variety of different learning outcomes for students from different backgrounds;(2) Choose cases intentionally; (3) Integrate student learning activities, supports, and materials; (4) Evaluate students' learning experiences with cases; and (5) Engage in collaborative reflection to facilitate learning and improvement on instructional practices with casecentered teaching.This article offers readers a glimpse into case-centered teaching in action as three instructors and a facilitator collectively reflect on their use of cases in evaluation education. Although we had previously learned from and taught with cases, we approached casecentered teaching and learning with new insights based on the previous articles in this volume (Bourgeois et al., this issue; Ensminger et al., this issue; Kallemeyn et al., this issue; Linfield & Tovey, this issue; Montrosse-Moorhead et al., this issue). Most evaluation educators who teach with cases draw from their own experiences in designing and using cases in their teaching (Bourgeois et al., this issue). Rather than maintaining our teaching as a private, individual practice, we engaged in collaborative reflection on our practice (Smith et al., 2015). This article follows a tradition of various forms of self-study on evaluation (Boyce & McGowan, 2018;Chouinard & Boyce, 2017;He et al., 2021; van Draanen, 2017). The purpose of this article is to share our collective actions regarding how to implement case-centered teaching in evaluation courses and how it shapes learning experiences.

show abstract

Evaluation criteria for artificial intelligence

Montrosse‐Moorhead

2023

New Drctns Evaluation

View full text Add to dashboard Cite

Criteria identify and define the aspects on which what we evaluate is judged and play a central role in evaluation practice. While work on the use of AI in evaluation is burgeoning, at the time of writing, a set of criteria to consider in evaluating the use of AI in evaluation has not been proposed. As a first step in this direction, Teasdale's Criteria Domains Framework was used as the lens through which to critically read articles included in this special issue. This resulted in the identification of eight criteria domains for evaluating the use of AI in evaluation. Three of these criteria domains relate to the conceptualization and implementation of AI in evaluation practice. Five criteria domains are focused on outcomes, specifically those stemming from the use of AI in evaluation. More work is needed to further identify and deliberate possible criteria domains for AI use in evaluation.

show abstract

Evaluative reasoning in public-sector evaluation in Aotearoa New Zealand: How are we doing?

Cited by 12 publications

References 29 publications

The overall evaluative conclusion as an element of evaluation practice

The overall evaluative conclusion as an element of evaluation practice

What does implementing case‐centered teaching of evaluation look like in practice?

Evaluation criteria for artificial intelligence

Contact Info

Product

Resources

About