This article reports the results of a meta-evaluation of 30 publicly accessible evaluation reports written or commissioned by 20 New Zealand public-sector agencies during the period 2010-2013 to understand how evaluative reasoning is being practised in Aotearoa New Zealand. The reports were examined to find evidence of five key elements of evaluative reasoning, namely, evaluative objectives or questions, criteria or other comparator(s), defined standards, a warranted argument, and an evaluative conclusion or judgement. Only eight of the evaluation reports had evidence of all five elements. While the focus of the meta-evaluation was on the presence of the five elements (not their quality) and the report sample is not representative, the study provides an interesting snapshot of evaluative practice in Aotearoa New Zealand. The findings suggest there may be scope to improve evaluative reasoning practice.

The turbulent and fluid environment in which we find ourselves due to the COVID-19 pandemic requires evaluative responses that facilitate learning, adaptation, and timeliness. This article examines the last of these—the need for timely evaluative information. Such information requires evaluators and their clients making trade-offs between what is desirable and what is feasible in a constrained time frame. Applying a light-hearted analogy—skateboard, pushbike, quad bike—three different evaluative approaches and the trade-offs that each involves are described. The notion of adequacy for purpose is then examined in terms of two dimensions: the level of certainty of the evidence that is required by the client, and the level of confidence required by the client in the evaluative claim/conclusion. The article demonstrates the need for evaluators to ensure their clients and other users of evaluative information understand the level of confidence and certainty that can be placed in it.

A new intervention that is designed to be adaptive presents a challenge for evaluators when developing a logic model at the start of a government policy evaluation. Our task became more problematic when the early data collection identified that participant groups had differing ideas about what the policy was intended to achieve. Participants also tended to ‘deconstruct’ the policy and focus only on the parts for which they were responsible. Realising that traditional logic modelling (that is, a linear approach to describing policy inputs and outcomes developed at the beginning of the evaluation) would provide a simplified picture of the policy, we decided to ground the logic in the evidence. This approach provided a more in-depth understanding of how the policy components interact with each other. Our logic became a vehicle to reconceptualise the policy through presenting an alternative perspective to understanding and communicating how the policy works on the ground.

This method note presents Q methodology as a useful tool for evaluators to add to their practice toolbox. Q methodology, which involves both quantitative and qualitative techniques, can help researchers and evaluators systematically understand subjectivity and the communicability of opinions and perspectives. We first provide an overview of Q methodology, followed by a brief summary of how evaluators are using Q, and an explanation of the steps for implementing Q methodology. Either by itself or with other methods, the potential uses of Q methodology in evaluation are diverse. For practical demonstration, we describe how Q methodology was used in a recent evaluation in the UK to understand stakeholder subjectivity within the program. We then reflect upon the pros and cons of using Q in program evaluation, concluding that it constitutes a worthwhile tool for evaluating complex programs.

No abstract