So What Do You Do? Exploring Evaluator Descriptions of Their Work

Abstract

To progress as a field, evaluators must be able to describe their work to nonevaluators, and to persuade an audience they have something unique and valuable to offer. Yet anecdotal reports suggest many evaluators have difficulty describing their work. Through 42 interviews with practicing evaluators, we aimed to better understand evaluators’ experiences discussing their work, examining how evaluators describe their work to nonevaluators and the perceived challenges they face doing so. Findings suggest there are common threads linking evaluator descriptions of their work, with many evaluators presenting evaluation as a tool to help social programs, policies, and organizations. And, although many evaluators described layperson reactions that centered on confusion, disinterest, and fear, others also described experiences characterized by understanding, interest, and gratitude. Strategies for enhancing communication in this domain are described.

What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research

“…Overall, this study provides some insight into the varied ways in which evaluators and researchers define evaluation. Similar to the study by Mason and Hunt (2018), this study demonstrates that most evaluators, as defined in this study by self-identifing primarily as an evaluator, define evaluation in terms of the purpose of the evaluation (i.e., for rendering a value judgment or for learning, program improvement, or decision-making) and some also describe the process of evaluation (i.e., in terms of the methodology, whether outcomes or processes are examined, or participation of stakeholders). However, there were differences between evaluators and researchers in whether processes or purposes were mentioned or prioritized and what type of processes and purposes were described.…”

“…Other research on free associations with the word "evaluation" found that students most associated evaluation with an appraisal whereas stakeholders were more likely to associate evaluation with improvement (Schultes, Kollmayer, Mejeh, & Spiel, 2018). Furthermore, Mason and Hunt (2018) There may be more dire consequences for a lack of a single definition of evaluation. With so many definitions of evaluation existing, one could go so far as to say that anything is evaluation.…”

“…. The coding scheme for the definition of evaluation question was adapted from Mason and Hunt (2018). Specifically, each definition provided by participants was coded for the process (i.e., whether a methodology is discussed, whether the process or outcomes of a program are described, whether stakeholders are involved in the process, and an 'other' category) and purpose (i.e., whether the purpose of evaluation is to provide a value judgment such as the effectiveness of a program or whether the purpose is to help programs improve) of evaluation.…”

“…What is evaluation? Perspectives of how evaluation differs (or not) from research Evaluators, emerging and experienced alike, lament how difficult it is to communicate what evaluation is to non-evaluators (LaVelle, 2011;Mason & Hunt, 2018). This difficulty in communicating what evaluation is stems partly from the field of evaluation having identity issues (Castro, Fragapane, & Rinaldi, 2016), leading to difficulty coming to a consensus of the definition of evaluation (Levin-Rozalis, 2003).…”

What is evaluation? Perspectives of how evaluation differs (or not) from research

“…with the word "evaluation" found that students most associated evaluation with an appraisal whereas stakeholders were more likely to associate evaluation with improvement (Schultes, Kollmayer, Mejeh, & Spiel, 2018). Furthermore, Mason and Hunt (2018) examined how evaluators describe evaluation to others and found that most evaluators tended to emphasize evaluation's purpose rather than its process, but would also vary their definitions depending on contextual factors such as a person's education, prior knowledge of evaluation, and their role in the organization. This research suggests a lack of a formal definition may make it difficult to communicate with our stakeholders or young and emerging evaluators (YEEs) what is evaluation.…”