Meaningful Evaluation: a holistic and systemic approach to understanding and assessing outcomes

Abstract

The concept and practice of Social Impact and Social Impact assessment has developed and matured over the last decade. Despite this growth, confusion still exists with respect to definitions of social impact as well as which tools and frameworks are most appropriate. Central to the concept and practice of social impact is a focus on outcomes. This paper argues that a discernible albeit problematic pattern and approach has emerged in outcomes measurement within the social impact field. After briefly reviewing some of the key problems with how outcomes measurement is practised, it presents some recent approaches from the evaluation field that attempt to address some of these concerns before introducing a new approach to understanding and evaluating outcomes – Meaningful Evaluation (ME). A pilot Meaningful Evaluation of volunteer ethics teachers in the Primary Ethics program in NSW is used to illustrate the approach. ME combines the Map of Meaning (MoM) with insights from next generation evaluation approaches to understand and assess outcomes. Informed by appreciative inquiry perspectives, it provides a means to bridge the divide between positivist and interpretivist approaches in evaluation. Key to ME is the assumption that it is more likely that immediate outcomes lead to medium and long-term outcomes (changes in behaviour) that are sustainable and lead to impact if participants experience program interventions as ‘meaningful’. Meaning is an important internal outcome that is essential if longer-term external outcomes are to occur. It also shifts the focus to capture unintended outcomes, key to developing holistic and systemic rather than linear and mechanistic Theories of Change.

Challenges of impact measurement in an emergent social economy

“…On the other hand, in the academic literature 'next generation' evaluation approaches have emerged, for example, the approach of Meaningful Evaluation (ME), which deal with complex social phenomena in a holistic and systemic manner and which is based on the Map of Meaning (MoM) model (Zappalà, 2020). As meaning is a subjective phenomenon, ME approach acknowledges the importance of focusing on the subjective experience of participants and their active participation in any program evaluation, although it is also applicable with more positivist approaches (Zappalà, 2020, p. 130).…”

Student and teacher outcomes from participating in a Philosophy for Children program: Volunteer ethics teachers’ perspectives

“…The findings reported in this paper come from a broader qualitative research study that sought to explore the experiences of volunteer ethics teachers in the Primary Ethics program in several schools in the greater Sydney area of NSW, Australia (Zappalà 2019;2020). 3 As part of the preliminary research, relevant pre-existing qualitative and quantitative data collected by Primary Ethics were analysed using the Meaningful Evaluation framework which acknowledges the importance of focusing on the extent to which participants experience the various activities and outcomes of a program as 'meaningful'.…”

“…3 As part of the preliminary research, relevant pre-existing qualitative and quantitative data collected by Primary Ethics were analysed using the Meaningful Evaluation framework which acknowledges the importance of focusing on the extent to which participants experience the various activities and outcomes of a program as 'meaningful'. This enables the identification and assessment of the sustainability of any outcomes experienced, captures the inner process of change, as well as any unintended consequences of participation in a program (Zappalà 2020).…”

“…All interviews were conducted in person at a location convenient to the volunteer, usually the school where they taught ethics, their home or a local café. The interview schedule, also based on the Meaningful Evaluation framework, covered a range of questions regarding ethics teachers' overall experience of the program, including their motivations, whether teaching ethics had led to any moral or personal growth, their sense of connection to others, their ability to be creative in teaching, and the extent to which it provided an opportunity to make a positive difference to the students and society (Zappalà 2020).…”

“…What kinds of engagement or interactions are more effective? Were there any unintended outcomes from participating in the program (Zappalà 2020)? This paper examines the impact of P4C from the teachers' perspective.…”