Developing evaluation criteria using an ethical lens

Roorda, Mathea; Gullickson, Arlen R.

doi:10.1177/1035719x19891991

Cited by 12 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scriven’s (2016) primary concerns are with how evaluators could deal more directly and thoughtfully with the ethical issues that arise within the contexts and initiatives being evaluated. While we agree with this and others’ calls to examine ethical issues within interventions and their contexts (Levin-Rozalis, 2014; Roorda and Gullickson, 2019; Schwandt and Gates, 2021), we explore a different yet related issue: the ethical basis for evaluation as a form of knowledge co-design and co-production for systems transformation (Page et al, 2016). In other words, what ethical issues arise within the practices of determining the purpose(s) of an evaluation, framing the scope of work of an evaluation, making decisions about what data and evidence should be gathered and how, centering some ways of knowing over others, and defending the legitimacy of evaluative processes and results.…”

Section: Introductionmentioning

confidence: 66%

Ethics of evaluation for socio-ecological transformation: Case-based critical systems analysis of motivation, power, expertise, and legitimacy

Gates

Page²,

Crespo

et al. 2022

Evaluation

View full text Add to dashboard Cite

Evaluation that supports social, ecological, and governance systems change and transformation raises ethical questions about what and whose worldviews do and should ground evaluative processes. This article illustrates one approach to ethical analysis within a case study of the first phase of an initiative to co-create a monitoring, evaluation, and learning system. The case drew on the principles of Blue Marble Evaluation in partnership with local staff and Indigenous leaders of the Amazon Sacred Headwaters Initiative. The approach uses critical and relational systems thinking to examine the sources of motivation, power, knowledge, and legitimacy that influence and should influence an evaluation system. Results reframe typical early phase evaluation process work from a contractual agreement to a co-created ethical space that engenders the legitimacy of the evaluation process. Contributions include a conceptual framework and process for ethical analysis that could be adapted by others.

show abstract

Section: Introductionmentioning

confidence: 66%

Ethics of evaluation for socio-ecological transformation: Case-based critical systems analysis of motivation, power, expertise, and legitimacy

Gates

Page²,

Crespo

et al. 2022

Evaluation

View full text Add to dashboard Cite

show abstract

“…The evaluation standards mentioned previously do include guidelines on aspects like making evaluative reasoning 'explicit'. However the selection and definition of such criteria also needs to be justifiable (Roorda and Gullickson, 2019) or 'valid' and so this area could also benefit from explicit theorising in relation to evaluation validity.…”

Section: Evaluation Criteria and Standardsmentioning

confidence: 99%

What does it mean for an evaluation to be ‘valid’? A critical synthesis of evaluation literature.

Macklin¹,

Gullickson²

2022

Preprint

View full text Add to dashboard Cite

Validity is both a critical foundation for, and widely contested concept in the field of program evaluation. Given the wide and divergent treatment of the term, this article sought to enhance the field’s understanding and application of validity by systematically identifying, collating and integrating existing conceptions of validity. A critical interpretive synthesis approach was taken to combine the best elements of systematic literature searching with integrative review. We found that conceptions of validity could be roughly collated into three categories pertaining to the evaluation research, evaluative judgement and reporting, and beyond the evaluation proper. The result was a wide-ranging map of validity concepts against the various stages of the evaluation process. This made clear several gaps in the validity thinking, resulting in the development of comprehensive but not yet complete framework that weaves together related aspects identified into a cohesive whole. It is our hope that this initial iteration of a cohesive validity framework will be the starting point for a cooperative effort to improve professional evaluation practice.

show abstract

“…Impact evaluation is one of those perspectives, focusing on the causality underlying the program -what is the program trying to create, does it actually accomplish its intents, and to what extent? It comes out of the deontological ethical tradition, focused on the greatest good for the greatest number (Roorda & Gullickson, 2019). To address the critique of that ethical tradition, good impact evaluation considers not only what has happened as a result of the program, but the value of those effects for different stakeholder groups (Pawson & Tilley, 2014) and any side effects or side impacts (Scriven, 1991;.…”

Section: The Role Of Impact Measurementmentioning

confidence: 99%

Developing an impact framework for Science Gallery Network: Final report

Mahat¹,

Morrow²,

Long³

et al. 2022

View full text Add to dashboard Cite

The aim of this project was to develop an impact framework for the Science Gallery Network (SGN). This work was commissioned by the Science Gallery International (SGI). The SGN has eight member organisations across four continents: Dublin, London, Melbourne, Bengaluru, Detroit, Rotterdam, Atlanta and Berlin. Whilst the network consistently sees unprecedented levels of accomplishment by its members, a testimony to their capacity, innovation and vision, the SGN does not have a systematic way to measure and monitor this impact. An impact framework that can assist with understanding and reporting the value of this impact will provide important recognition that the SGN has achieved what it sets out to do— bringing science, art, technology and design together to deliver world-class educational and cultural experiences for young people. This report details the robust consultation approach that was undertaken by the University of Melbourne’s project team—one that included a desktop review, focus group discussions, surveys and interviews—to ensure multiple perspectives were gathered on what could be considered a multi-faceted concept. The desktop review provided a thorough review and an environmental scan of the impact literature and its measurement. In addition, the focus group discussions and interviews provided a rich understanding of what ‘good impact’ means for the SGN and the implications of this to the measurement of impact outcomes. Five key recommendations are provided and summarised. Note that these key recommendations should be taken as a point of departure for further in-depth consultation throughout the wider SGN.

show abstract

Developing evaluation criteria using an ethical lens

Cited by 12 publications

References 29 publications

Ethics of evaluation for socio-ecological transformation: Case-based critical systems analysis of motivation, power, expertise, and legitimacy

Ethics of evaluation for socio-ecological transformation: Case-based critical systems analysis of motivation, power, expertise, and legitimacy

What does it mean for an evaluation to be ‘valid’? A critical synthesis of evaluation literature.

Developing an impact framework for Science Gallery Network: Final report

Contact Info

Product

Resources

About