Integrating Big Data Into Evaluation: R Code for Topic Identification and Modeling

This article surveyed different emerging technologies (ET), in particular artificial intelligence, and their burgeoning application in the evaluation industry. Evidence suggests that evaluators have been relatively slow in adopting ET in their practice. However, more recent data suggest that ET adoption is increasing. This article then analyzed if, and how, ET affect the evaluation industry and evaluation practice. The article finds that program evaluation is one of several competing forms of knowledge production informing decision‐making, particularly in the government and not‐for‐profit sectors. Therefore, evaluation faces a number of challenges stemming from ET. In this article, it is argued that evaluators must, albeit critically, embrace ET. Most likely, ET will complement evaluation practice and, in some instances, displace human tasks.

Section: Resultsmentioning

confidence: 99%

Section: Will Et Affect Evaluation As An Industry?mentioning

confidence: 99%

Disrupting evaluation? Emerging technologies and their implications for the evaluation industry

Nielsen

2023

“…Recent examples of NLP in use in evaluation studies include Anglin et al. (2021); Cintron and Montrosse‐Moorhead (2022); Roy and Rambo‐Hernandez (2021), to name a few.…”

Section: Examples Of Ai Typesmentioning

confidence: 99%

“…Voice controlled digital assistants, like Apple's Siri (Apple Inc., 2023) or Amazon's Alexa (Amazon, 2023) use NLP, which is why these services become better at understanding speakers over time as they amass more and more audio data on which to train. Recent examples of NLP in use in evaluation studies include Anglin et al (2021); Cintron and Montrosse-Moorhead (2022); Roy and Rambo-Hernandez (2021), to name a few.…”

Section: Examples Of Ai Typesmentioning

confidence: 99%

A special delivery by a fork: Where does artificial intelligence come from?

Thornton

2023

In this article, I discuss the use of artificial intelligence (AI) in evaluation and its relevance to the evolution of the field. I begin with a background on how AI models are developed, including how machine learning makes sense of data and how the algorithms it develops go on to power AI models. I go on to explain how this foundational understanding of machine learning and natural language processing informs where AI might and might not be effectively used. A critical concern is that AI models are only as strong as the data on which they are trained, and evaluators should consider important limitations when using AI, including its relevance to structural inequality. In considering the relationship between AI and evaluation, evaluators must consider both AI's use as an evaluative tool and its role as a new subject of evaluation. As AI becomes more and more relevant to a wider array of fields and disciplines, evaluators will need to develop strategies for how good the AI is (or is not), and what good the AI might (or might not) do.

“…Presumably to remedy this state of affairs, scholars have advanced three related threads: Some have outlined arguments for why it is important to integrate AI in evaluation (Bamberger, 2016; Hejnowicz & Chaplowe, 2021; Leeuw, 2020; York & Bamberger, 2020). Examples of potential uses of techniques grounded in one of AI's branches are being published, which offer case examples from which evaluation pracitioners can learn (see, e.g., Cintron & Montrosse‐Moorhead, 2021; Raveh et al., 2020; Roy & Rambo‐Hernandez, 2021). Others have highlighted critical questions about the use of AI in evaluation (Leeuw, 2020; Picciotto, 2020).…”

Section: The Importance Of Evaluative Criteria For Artificial Intelli...mentioning

confidence: 99%

Evaluation criteria for artificial intelligence

Montrosse‐Moorhead

2023

Criteria identify and define the aspects on which what we evaluate is judged and play a central role in evaluation practice. While work on the use of AI in evaluation is burgeoning, at the time of writing, a set of criteria to consider in evaluating the use of AI in evaluation has not been proposed. As a first step in this direction, Teasdale's Criteria Domains Framework was used as the lens through which to critically read articles included in this special issue. This resulted in the identification of eight criteria domains for evaluating the use of AI in evaluation. Three of these criteria domains relate to the conceptualization and implementation of AI in evaluation practice. Five criteria domains are focused on outcomes, specifically those stemming from the use of AI in evaluation. More work is needed to further identify and deliberate possible criteria domains for AI use in evaluation.