Evaluation as work in progress: Stories of shared learning and development

Abstract

Culture change programs that attempt to identify, challenge, and change inequities and discrimination represent major challenges for evaluators: these are shifting sands, subject to many external factors that cannot be accounted for or measured using 'objective' measures. Yet, effective, methodologically coherent evaluation of such projects is essential for both ethical and pragmatic reasons. This article reports on the evaluation of two programs from Victoria, Australia, that use sport as a vehicle for building cultures of respect. The methodology that underpinned both evaluations is described and the authors discuss the challenges (and benefits) experienced during the evaluation process. These challenges and benefits can be summarized under two main headings: relationships and expectations. Effective relationships based on trust led to open and frank communication, and this was essential to manage the multiple and often contradictory expectations that often accompany large, complex evaluation projects.

Methodological Approaches to Assessing the Quality of an Organisation’s Performance

“…In data analysis (5), it is important to draw on theoretical and other strategies, apply selected data analysis methods, use quantitative and qualitative data, and analyse and explore new data in isolation from interpretations. In the data processing phase (6), it is important to define the audience, create textual and visual material, and provide sufficient evidence to allow the study reader to draw their own conclusions.…”

“…In the evaluation process itself, 'human instruments' are essential. The fact that this process is highly complex and requires a great deal of expertise and insight is demonstrated by studies carried out in different countries (e.g., Fletcher and Dyson [6]; Grissom, Blissett and Mitani [7], etc.). Thus, in line with the fourth-generation evaluation paradigm, the purpose of the evaluation process is not only to collect data, but also to engage the organisational community, to foster accountability, and to bring about a positive change in organisational culture.…”