Automated coding of student chats, a trans-topic and language approach

Araujo, Adelson de; Papadopoulos, Pantelis M.; McKenney, Susan; Jong, Ton de

doi:10.1016/j.caeai.2023.100123

Cited by 6 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a concrete example, a recent study evaluated the passage generation with LLMs in the Progress in International Reading Literacy Study (PIRLS) (Bezirhan & von Davier, 2023). The study employed the released PIRLS passages as input and prompted GPT3.5 to generate a similar reading passage at the fourth‐grade reading level but with a different topic.…”

Section: Impacts and Implications For Assessmentmentioning

confidence: 99%

“…Moreover, when integrating the multimodal capability of generative AI, items with graphics in multiple languages can be generated at high quality. This greatly eases the process of developing such items in large‐scale international survey assessments such as the Programme for International Student Assessment (PISA), International Computer and Information Literacy Study (ICILS), International Civic and Citizenship Education Study (ICCS), and Programme for the International Assessment of Adult Competencies (PIAAC) (Bezirhan & von Davier, 2023; von Davier et al., 2023). Figure 2 shows a science item in multiple languages and with images (Jung et al., 2022).…”

Section: Impacts and Implications For Assessmentmentioning

confidence: 99%

See 1 more Smart Citation

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

Hao,

von Davier,

Yaneva

et al. 2024

Educational Measurement

View full text Add to dashboard Cite

The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have unveiled a wealth of opportunities and challenges in assessment. Applying cutting‐edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely, these innovations raise significant concerns regarding validity, reliability, transparency, fairness, equity, and test security, necessitating careful thinking when applying them in assessments. In this article, we discuss the impacts and implications of LLMs and generative AI on critical dimensions of assessment with example use cases and call for a community effort to equip assessment professionals with the needed AI literacy to harness the potential effectively.

show abstract

Section: Impacts and Implications For Assessmentmentioning

confidence: 99%

Section: Impacts and Implications For Assessmentmentioning

confidence: 99%

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

Hao,

von Davier,

Yaneva

et al. 2024

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…The preference for this method is based on the knowledge that GPT performs better after being provided with clear and diverse instructions during the text generation process (Brown et al., 2020). Following the instructions that were provided for each sentence during the generation process, the one‐shot learning method was chosen to achieve better performance (Bezirhan & von Davier, 2023). The parent item (Table 1) was presented as an example for GPT.…”

Section: Text Analysis Cognitive Modelmentioning

confidence: 99%

“…(2022) used GPT‐3 to create interactive reading passages involving human reviewers. Bezirhan and von Davier (2023) also sought expert opinions to assess the quality of the texts generated with GPT. In the current study, the evaluation process was carried using outcomes from GPT, SME and field tests.…”

Section: Text Analysis Cognitive Modelmentioning

confidence: 99%

See 1 more Smart Citation

Using OpenAI GPT to Generate Reading Comprehension Items

Sayin,

Gierl

2024

Educational Measurement

View full text Add to dashboard Cite

The purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template‐based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT‐3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject‐matter experts. The generated items were also administered to a sample of 1,607 Grade‐8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.

show abstract