When and How Qualitative Methods Provide
            Credible and Actionable Evidence : Reasoning With Rigor, Probity,
              and Transparency

Rallis, Sharon F.

doi:10.4135/9781483385839.n10

Cited by 9 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some have argued that evaluation is research and that there are no or few distinctions between the two ( Figure 1A). For example, some explicitly state that "evaluation is applied research…" (Barker et al, 2016;Hackbarth & Gall, 2005;Rallis, 2014), although sometimes adding caveats to those differences in later sentences. However, many recognize at least some differences between evaluation and research.…”

Section: Differences Between Evaluation and Researchmentioning

confidence: 99%

“…Furthermore, the similarity between related fields-auditing, management consulting, and especially social science research-exacerbates the issue of defining and communicating about evaluation. While some evaluators do not see a difference between applied social science research and program evaluation, stating simply that "evaluation is applied research" (e.g., Barker, Pistrang, & Elliott, 2016;Hackbarth & Gall, 2005;Rallis, 2014), others agree that although evaluation uses social science methodology it is distinct from social science research (Montrosse-Moorhead, Bellara, & Gambino, 2017). As a result of this lack of consensus within the field, the general public has a weak or "fuzzy" (Picciotto, 2011, p. 171) understanding of what evaluation is and does not recognize the distinctiveness of evaluation from research.…”

mentioning

confidence: 99%

See 1 more Smart Citation

What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research

Linnell

2020

American Journal of Evaluation

View full text Add to dashboard Cite

With a lack of consensus of what evaluation is within the field of evaluation, there is a difficulty in communicating to nonevaluators what evaluation is and how evaluation differs from research. To understand how evaluation is defined, both evaluators and researchers were asked how they defined evaluation and, if at all, differentiated evaluation from research. Overall, evaluators believed research and evaluation intersect, whereas researchers believed evaluation is a subcomponent of research. Furthermore, evaluators perceived greater differences between evaluation and research than researchers did, particularly in characteristics relevant at the beginning (e.g., purpose, questions, audience) and end (e.g., rendering value judgments, disseminating results) of studies. This study suggests that greater consensus on a definition of evaluation is needed to be able to distinguish the field and discipline of evaluation from related fields and to be able to communicate this information to nonevaluators.

show abstract

Section: Differences Between Evaluation and Researchmentioning

confidence: 99%

mentioning

confidence: 99%

What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research

Linnell

2020

American Journal of Evaluation

View full text Add to dashboard Cite

show abstract

“…Levin-Rozalis (2003) noted that in her professional practice, she frequently encounters a lack of understanding of the difference between research and evaluation, and it "is at the expense of evaluation" (p. 2). Wanzer (2021) found evidence of five different conceptualizations of the relationship between research and evaluation held by practicing evaluators (e.g., evaluation and research as a Venn diagram, evaluation as a type of applied research), which are also present in scholarly writing defining evaluation (Mathison, 2008;Mertens, 2014;Rallis, 2014;Scriven, 2008;Vedung, 2004). In Wanzer's study, while most AEA members were more likely than Division H (Research, Evaluation, and Assessment in Schools) American Educational Research Association members to consider themselves evaluators, only 62% conceived of evaluation-research relationship as being like a Venn diagram.…”

Section: Discussionmentioning

confidence: 99%

Evaluator Education Curriculum: Which Competencies Ought to Be Prioritized in Master’s and Doctoral Programs?

Montrosse-Moorhead

Gambino

Yahn

et al. 2021

American Journal of Evaluation

View full text Add to dashboard Cite

A budding area of research is devoted to studying evaluator curriculum, yet to date, it has focused exclusively on describing the content and emphasis of topics or competencies in university-based programs. This study aims to expand the foci of research efforts and investigates the extent to which evaluators agree on what competencies should guide the development and implementation of evaluator education. This study used the Delphi method with evaluators ( n = 11) and included three rounds of online surveys and follow-up interviews between rounds. This article discusses on which competencies evaluators were able to reach consensus. Where consensus was not found, possible reasons are offered. Where consensus was found, the necessity of each competency at both the master’s and doctoral levels is described. Findings are situated in ongoing debates about what is unique about what novice evaluators need to know and be able to do and the purpose of evaluator education.

show abstract

“…Developing evaluative questions and selecting methods that can provide data to answer those questions is one of the first steps in evaluation (Rallis, 2015). If the data collection methods match the evaluation purpose, and are implemented correctly, evaluations will produce credible and actionable evidence (Rallis, 2015). Every method has unique advantages and inherent challenges, which need to be considered before selecting them for data collection (Jones et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

Group Mapping in a 4-H Camp Mixed-Methods Evaluation

Koundinya

Klink

Skluzacek

et al. 2020

JYD

View full text Add to dashboard Cite

A week-long residential 4-H Agri-science youth camp had the goal of increasing understanding of agriculture as a system, however evaluating such a complex construct is difficult by adopting any one data-collection method. This paper presents the methods used in the evaluation of the camp and how these methods enhanced the overall program evaluation. The camp introduced 28 students, who were about 12 years of age from rural and urban counties of Wisconsin to the different aspects of the agricultural system, and promoted understanding of the connections among the various system components. The study utilized a mixed-methods triangulation design with group mapping (qualitative and quantitative), participant observation (qualitative), student reflections in the form of group presentations (qualitative), and retrospective post-then-pre surveys (quantitative) to measure and assess the anticipated outcomes. The qualitative assessment of the maps from Days 1 to 5 showed a significant improvement in students’ understandings of the interconnections of farm systems, which was validated by quantitative assessment of maps. Participant observation revealed that the camp provided students with a rich experiential learning experience and identified areas of improvement such as coordination of efforts between the different camp instructors. Student reflections and the end-of-camp survey indicated that the students increased their knowledge and skill levels as a result of attending the camp. The mixed-methods design used in this study serves as an example for designing better quality youth and Extension education program evaluations, as well as programs where the goal is to increase participants’ understandings of complex systems.

show abstract

When and How Qualitative Methods Provide Credible and Actionable Evidence : Reasoning With Rigor, Probity, and Transparency

Cited by 9 publications

References 13 publications

What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research

What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research

Evaluator Education Curriculum: Which Competencies Ought to Be Prioritized in Master’s and Doctoral Programs?

Group Mapping in a 4-H Camp Mixed-Methods Evaluation

Contact Info

Product

Resources

About