Editors' Notes

Azzam, Tarek; Evergreen, Stephanie D. H.

doi:10.1002/ev.20064

Cited by 2 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These kinds of discrete generative tasks with verifiable answers are well‐suited to AI applications. Tasking AI with producing more complex or richer qualitative narratives, however, can be less reliable (Azzam, 2023). Investigative journalists have demonstrated that generative AI may exhibit confusing responses, react inappropriately to prompts, or reflect the worst facets of the internet in its attempts to match human conversation (Roose, 2023).…”

Section: Ai and Evaluationmentioning

confidence: 99%

“…The most important consideration in machine learning models is this—what the machine learns depends on what you teach it. These models can only ever be as strong as the data on which they are trained and are subject to an array of validity concerns (Azzam, 2023). Training a machine learning model can exponentiate even very minute errors, particularly in supervised models where humans shape the algorithm's decision‐making.…”

Section: Limitations Of Ai That Evaluators Should Considermentioning

confidence: 99%

“…The most important consideration in machine learning models is this-what the machine learns depends on what you teach it. These models can only ever be as strong as the data on which they are trained and are subject to an array of validity concerns (Azzam, 2023).…”

Section: Limitations Of Ai That Evaluators Should Considermentioning

confidence: 99%

See 2 more Smart Citations

A special delivery by a fork: Where does artificial intelligence come from?

Thornton

2023

New Drctns Evaluation

View full text Add to dashboard Cite

In this article, I discuss the use of artificial intelligence (AI) in evaluation and its relevance to the evolution of the field. I begin with a background on how AI models are developed, including how machine learning makes sense of data and how the algorithms it develops go on to power AI models. I go on to explain how this foundational understanding of machine learning and natural language processing informs where AI might and might not be effectively used. A critical concern is that AI models are only as strong as the data on which they are trained, and evaluators should consider important limitations when using AI, including its relevance to structural inequality. In considering the relationship between AI and evaluation, evaluators must consider both AI's use as an evaluative tool and its role as a new subject of evaluation. As AI becomes more and more relevant to a wider array of fields and disciplines, evaluators will need to develop strategies for how good the AI is (or is not), and what good the AI might (or might not) do.

show abstract

Section: Ai and Evaluationmentioning

confidence: 99%

Section: Limitations Of Ai That Evaluators Should Considermentioning

confidence: 99%

See 1 more Smart Citation

A special delivery by a fork: Where does artificial intelligence come from?

Thornton

2023

New Drctns Evaluation

View full text Add to dashboard Cite

show abstract

“…Azzam's (2023) discussion of validity types and the ability of evaluators versus AI to support them.…”

Section: Proposed Criteria For Ai Use In Evaluationmentioning

confidence: 99%

“…The key question this criterion is centered on is: Are claims generated during an evaluation using AI valid or trustworthy? One aspect of this criterion is the validity or trustworthiness of inferences resulting from quantitative and/or qualitative AI methods, measurement, and analysis (Azzam, 2023; Sabarre et al., 2023; Tilton et al., 2023). Another consideration is whether and to what extent AI methods, measurement, and analysis are consistent with theory, best practices, principles, and standards (Azzam, 2023; Head et al., 2023).…”

Section: Proposed Criteria For Ai Use In Evaluationmentioning

confidence: 99%

Evaluation criteria for artificial intelligence

Montrosse‐Moorhead

2023

New Drctns Evaluation

View full text Add to dashboard Cite

Criteria identify and define the aspects on which what we evaluate is judged and play a central role in evaluation practice. While work on the use of AI in evaluation is burgeoning, at the time of writing, a set of criteria to consider in evaluating the use of AI in evaluation has not been proposed. As a first step in this direction, Teasdale's Criteria Domains Framework was used as the lens through which to critically read articles included in this special issue. This resulted in the identification of eight criteria domains for evaluating the use of AI in evaluation. Three of these criteria domains relate to the conceptualization and implementation of AI in evaluation practice. Five criteria domains are focused on outcomes, specifically those stemming from the use of AI in evaluation. More work is needed to further identify and deliberate possible criteria domains for AI use in evaluation.

show abstract

Editors' Notes

Cited by 2 publications

References 0 publications

A special delivery by a fork: Where does artificial intelligence come from?

A special delivery by a fork: Where does artificial intelligence come from?

Evaluation criteria for artificial intelligence

Contact Info

Product

Resources

About